Code to generate the figures for the manuscript "Robust representations learned in RNNs through implicit balance of expansion and compression" submitted to Nature Machine Intelligence. Information about authorship is currently withheld for the sake of double blind review.
The script figures_for_manuscript.py contains code to call plotting functions that generate the figures in the paper. This script primarily calls plots.py, which contains the code for generating the figures.
To run a faster version of the simulations, set FAST_RUN to True at the top of figures_for_manuscript.py. This runs only a single realization of the network for each set of hyperparameters. To use a version of the simulations that corresponds more closely with the plots in the paper, set FAST_RUN to False. This trains five networks for every set of hyperparameters, each with a different random number generator seed, and the variation is used to plot error bars. While this is sufficient to reproduce the results of the paper, the error bars in the paper use 30 network realizations. To simulate this, change the line "seeds = list(range(5))" to "seeds = list(range(30))" in figures_for_manuscript.py.
In the paper, the error bars for plots are generated by a custom modification made to the seaborn plotting library. This enables using a (shifted and reflected) beta distribution to fit the variation, which is a better model when the data are bounded (for example, accuracy is bounded above by 1). For simplicity, the code here is set to plot 95% confidence intervals (under the assumption of normally distributed error). If the individual lines are desired instead of error bars, set USE_ERRORBARS at the top of plots.py to False.
When the neural networks are trained, the weights through training are saved in a directory in the output folder, and this directory location is noted in the table output/output_table.csv. This table maps the model output directory to the set of parameters used to train the model (i.e. those parameters passed to initialize_and_train.initialize_and_train). This mapping of parameters and save locations is handled by model_output_manager.py. The next time initialize_and_train.initialize_and_train is called, it will automatically check for a previous run that matches that set of parameters, and load the model if it finds it. To disable this behavior, either keep the directory "output/"" empty, or set rerun=True in the parameters passed to initialize_and_train.initialize_and_train. The variable CLEAR_PREVIOUS_RUNS at the top of figures_for_manuscript.py can also be set to true, which clears out the "output/" folder before training. Codeocean automatically removes all saved data after finishing, but if run outside of code ocean the saved parameters should remain in the directory data/output.
There are a variety of hyperparameter options that can be set, including looking at the layers of a feedforward network instead of the timesteps of an RNN, or looking at other RNN models (the parameter 'network' specifies this). These hyperparameters haven't necessarily been exhaustively tested and exceptions may be thrown if they are changed.
The parameter 'g_radius' in the code controls the level of chaos of the initialized RNN, and corresponds with the parameter gamma in the manuscript.
The figures for the pca snapshots are labelled by timepoint / layer, so "Xdim_200_snapshot_0_tanh_cce" is the snapshot at time t=0. "Xdim_200_snapshot_-1_tanh_cce" is the pca plot of the inputs.