Researchers from Microsoft Research have developed an interactive system for real-time analysis of deep learning training called TensorWatch.
The new tool was designed to be a flexible visualization tool for data science and deep learning projects and that will work in Jupyter notebooks. According to its creators, Tensorwatch has the unique capability to execute user queries during a live training process. With this feature, it overcomes the limitations of current training monitoring and visualization tools that require logging the data and do not offer the flexibility to modify what information is visualized.
TensorWatch is also extensible and users can render real-time information in multiple formats and build their own visualizations. In order to use the tool, it is necessary and sufficient to register a TensorWatch stream, which writes to a file or streams directly to a Jupyter notebook. Everything within TensorWatch is built as a stream. This allows users to create arbitrary data flow graphs and provides a lot of flexibility in how data is stored or visualized.
The tool is still under heavy development and its implementation is open-source and available on Github. More in detail about the architecture and the features of TensorWatch can be read in the paper. Tutorials on how to start with TensorWatch are also available in the Github repository.