# Output data model Every file GuPPy writes to disk: where it lands, what its name means, and what is inside it. `store_id`, `store_label`, `recording_site`, `session_folder`, `input_root_folder`, `output_root_folder`, `run` and `run_folder` are used here as the [Glossary](glossary.md) defines them. --- ## Where outputs go Every output of a run lives in a single directory, the run folder. Each session's path below the `input_root_folder` is mirrored inside the `output_root_folder`, and the run folders are created in that mirror: ``` //output_/ ``` With roots of `/data` and `/data_analysis`, a session at `/data/subject1/session1` gets its run folders at `/data_analysis/subject1/session1/output_1`. Both roots are set in the **Root Folder Selection** card and both are required. Nothing is written into a session folder except a custom event you import yourself, which Step 1 discovers beside the acquisition files. `` is either the name you type in the Label Stores GUI or, when you leave it to GuPPy, the lowest integer for which no such directory exists yet — `output_1` on the first run, `output_2` on the second. Re-running Step 1 over an existing run folder with the overwrite option deletes its entire contents first. Each run folder is self-contained, so one session can have several runs under different parameters — see [Comparing Two Parameter Sets](../tutorials/compare_parameters.md). Ticking **Output root folder is the same as the input root folder** points both roots at one folder, so each session mirrors onto itself and its run folders are created inside the session folder. Run folders written before version 2.0.0-beta4 are named `_output_`. They are still found and still opened, and they occupy their run name so a new run never collides with one; GuPPy never creates the older name, and reading it is deprecated and will be removed in a future release. Three further directories can appear: | Directory | Location | Written by | |-----------|----------|------------| | `cross_correlation_output/` | inside a run folder | Step 4, when **Compute Cross-correlation** is enabled | | `psth_significance_output/` | inside a run or group folder | Step 4 and Group Analysis, when **Compute PSTH Significance?** is enabled | | `_group/` | in the destination directory you pick | the Label Groups step (definition) and the Group Analysis step (results) | A group directory is not inside any session folder. It is created in the destination directory chosen on the Label Groups page. --- ## File formats GuPPy's two HDF5 extensions are **not** interchangeable: `.hdf5` and `.h5` are written by different libraries and have to be read by different ones. [Custom Plots from GuPPy Outputs](../tutorials/custom_plots.md) works through loading each of them in Python. ### `.hdf5` Raw HDF5 written with h5py, following GuPPy's convention of one file per store with several named datasets inside it. Array datasets are one-dimensional and chunked; scalars (a store's name, its sample count) are zero-dimensional datasets. **No HDF5 attributes are set anywhere** — `sampling_rate` and the other scalars are datasets, not attributes. ### `.h5` A pandas DataFrame written with `DataFrame.to_hdf`, holding exactly one DataFrame under the key `df`, and read back with `pandas.read_hdf` rather than h5py. These are the PSTH, peak/AUC, transient-summary, binned-metrics, covariate, cross-correlation and tonic tables. `to_hdf` stores the frame in pandas' fixed format, which mirrors its in-memory layout: the column labels in `axis0`, the row index in `axis1`, and the values in one or more `block_values` arrays with columns grouped by dtype. None of that is hidden from h5py, but it is the frame taken apart rather than the assembled table, and `axis0` order does not follow the block grouping — putting it back together by hand means matching each block against its own `block_items`. ### `.csv` Flat text. Inside a run folder: the store mappings (`storesList.csv`, `combine_storesList.csv`), the synthetic control trace `cntrl.csv`, tables that also exist as an `.h5` (peak/AUC, transient frequency and amplitude, binned metrics, binned covariates and covariate correlations), and the two tables written as CSV only — `transientsOccurrences_.csv` and `tonic_epochs_.csv`. Imported custom events are CSV too. ### `.npy` A raw NumPy array, used only for the artifact keep-windows in `coordsForPreProcessing_.npy`. ### `.json` Provenance and configuration: `GuPPyParamtersUsed.json` and `.npm_params.json`. ### `.yaml` Plain text, written only by Step 6: the NWB metadata overlay `nwb_metadata.yaml`. It is the authoritative artifact behind the metadata form, and is meant to be read, edited and copied between sessions by hand. ### `.nwb` HDF5 written through pynwb, holding the whole session — raw traces, events and every GuPPy product — as typed NWB objects rather than named arrays. Written only by Step 7. The file carries the `ndx-fiber-photometry`, `ndx-ophys-devices` and `ndx-guppy` schema extensions inside it, so it stays readable without them installed. See [Fiber photometry data in NWB](../explanation/nwb.md). --- ## Filename vocabulary The placeholders used in the filename patterns: | Placeholder | Meaning | Example | |-------------|---------|---------| | `` | The store name as it appears in the acquisition system's own files | `Dv1A`, `405R`, `Sample_Signal_Channel` | | `` | The analytic label you assigned in Step 1 | `signal_DMS`, `control_DMS`, `port_entries` | | `` | A recording site — the suffix shared by a `signal_`/`control_` label pair | `DMS`, `region` | | `` | A preprocessed trace name, always `z_score_` or `dff_` | `z_score_DMS` | | `` | The metric without its recording site | `z_score`, `dff` | | `` | The `store_label` of an event store | `port_entries`, `RewardPort` | Because `` already ends in the recording site, filenames that combine an event with a metric carry the site twice: `port_entries_DMS_z_score_DMS.h5` is the PSTH of the event `port_entries` at site `DMS`, computed on the `z_score_DMS` trace. Backslashes and forward slashes in a store or event name are replaced with underscores before the name reaches a filename, so a TDT store named `PAB/` produces files beginning `PAB_`. --- ## Step 1: Label your channels | File | Contents | |------|----------| | `storesList.csv` | The store id → store label mapping for the run | | `.npm_params.json` | Neurophotometrics decomposition settings (NPM sessions only) | **`storesList.csv`** is a headerless CSV of exactly two rows and one column per store. Row 0 holds the `store_id` values, row 1 the `store_label` values you assigned, column by column. Every later step reads it, and a run folder without it is not a valid run. ``` Dv1A,Dv2A,PrtN control_DMS,signal_DMS,port_entries ``` **`.npm_params.json`** is written only for Neurophotometrics sessions and holds the settings Step 2 uses to demultiplex the interleaved channels: `npm_split_events`, `npm_time_unit` (always stored resolved, never null), `npm_timestamp_column_name` and `noChannels`. --- ## Step 2: Load the raw data | File | Contents | |------|----------| | `.hdf5` | One file per store, holding that store's raw samples and timestamps | These files are named after the **store id**, not the store label — `Dv1A.hdf5`, not `signal_DMS.hdf5`. Step 3 writes a second, separate set of files named after the store labels; see [Two sets of per-store HDF5 files](#two-sets-of-per-store-hdf5-files). Which datasets a file carries depends on the acquisition format and on whether the store is a continuous trace or an event stream. Event stores have timestamps but no `data`. | Dataset | Shape | Contents | Written by | |---------|-------|----------|------------| | `timestamps` | 1-D float | Sample or event times on the acquisition clock | every format | | `data` | 1-D float | The samples (continuous stores only) | every format | | `sampling_rate` | 0-D float (TDT, NWB); 1-D float, length 1 (CSV, Doric, NPM, pyPhotometry) | Samples per second (continuous stores only) | every format | | `store_id` | 0-D string | The store's own name | TDT, NWB | | `npoints` | 0-D int | Samples per acquisition block | TDT, NWB | | `channels` | 1-D int | Per-block channel number | TDT | For TDT stores, `data` is longer than `timestamps`: the acquisition writes samples in blocks of `npoints` and timestamps one per block, so the two arrays only line up after Step 3 has expanded the timebase. For every other format they are the same length. TDT epoc stores whose strobe values encode several distinct behaviors are split at discovery time into one store per strobe value, named by appending the formatted value to the store id — `PAB/` with strobe `16` becomes `PAB16.hdf5`. Each split store gets its own file, with `timestamps` filtered to that strobe value. --- ## Step 3: Preprocess the signal | File | Contents | |------|----------| | `timeCorrection_.hdf5` | The corrected timebase for one recording site | | `.hdf5` | The trimmed and filtered samples for one labeled store | | `_.hdf5` | Event timestamps mapped onto the corrected timebase | | `z_score_.hdf5` | The z-scored trace | | `dff_.hdf5` | The ΔF/F trace | | `cntrl_sig_fit_.hdf5` | The control channel after fitting it to the signal | | `cntrl.hdf5` | Placeholder store, non-isosbestic recordings only | | `combine_storesList.csv` | Merged store mapping, **Combine Data?** runs only | **`timeCorrection_.hdf5`** carries five datasets: `timestampNew` (the corrected timestamp array, the timebase every other trace at this site is on), `correctionIndex` (the index array applied to the raw timestamps to produce it), `sampling_rate` (length-1 array), `timeRecStart` (the raw first timestamp, before correction) and `recordingStart` (the instant the corrected timebase treats as the recording start — `0.0` for TDT, the raw first timestamp for every other format). **`.hdf5`** holds a single dataset `data`: the store's samples after the warm-up seconds have been trimmed and the moving-average filter applied. One file per labeled store, so `signal_DMS.hdf5` and `control_DMS.hdf5`. **`_.hdf5`** holds a single dataset `ts`: the event's timestamps on the corrected timebase. There is one file per (event, recording site) pair, each carrying that site's own corrected timestamps — an event store labeled `port_entries` in a two-site recording produces both `port_entries_DMS.hdf5` and `port_entries_DLS.hdf5`. Step 4 rewrites `ts` in place with the subset of timestamps the PSTH actually used. **`z_score_.hdf5`, `dff_.hdf5` and `cntrl_sig_fit_.hdf5`** each hold a single dataset `data`. `cntrl_sig_fit_.hdf5` is the trace the preprocessing and artifact-removal views overlay on the signal. **`cntrl.hdf5` and the synthesized control** appear when **Isosbestic Control Channel?** is off. GuPPy first copies the signal store's file to `cntrl.hdf5` as a placeholder and appends a `cntrl` → `control_` column to `storesList.csv`, then writes the synthetic exponential-decay control it fits to the signal into `control_.hdf5` under the key `data`. In this mode `control_.hdf5` is a GuPPy-generated trace, not a recorded one. **`combine_storesList.csv`** appears only when **Combine Data?** is on. It has the same two-row shape as `storesList.csv` and holds the union of the store mappings across all sessions in the combined group. The combined traces themselves are written into the first run folder of the group, overwriting its `timeCorrection_.hdf5`, `.hdf5` and `_.hdf5` files. ### Two sets of per-store HDF5 files A run folder ends up with two overlapping sets of HDF5 files. Step 2 writes one per **store id** (`Dv1A.hdf5`) holding the raw samples straight from the acquisition files. Step 3 writes one per **store label** (`signal_DMS.hdf5`) holding the same channel after trimming and filtering. Nothing renames or replaces the Step 2 files; both sets sit side by side, and `storesList.csv` is what connects them. --- ## Artifact windows and removal *Written by: Select Artifact Windows and Remove Artifacts (both optional, between Steps 3 and 4).* | File | Contents | |------|----------| | `coordsForPreProcessing_.npy` | The periods of the recording to **keep** | **`coordsForPreProcessing_.npy`** is a float array of shape `(2M, 2)`. Column 0 holds the keep-window bounds interleaved as `[start_0, end_0, start_1, end_1, …]`; column 1 is an unused placeholder of zeros. The file records what survives, not the artifact periods you marked in the GUI. A recording site with no marked artifacts gets no file at all, and the pipeline then treats the whole recording as one keep window. Remove Artifacts does not create new files. It rewrites the Step 3 outputs in place: `data` in each `.hdf5`, and `ts` in each `_.hdf5`. The `concatenate` method additionally rewrites `timestampNew` and sets `recordingStart` to `0.0` in `timeCorrection_.hdf5`. See [Remove artifacts from a recording](../how-to/artifact-removal.md). --- ## Tonic analysis *Written by: Tonic Analysis (optional, between Remove Artifacts and Step 4).* | File | Contents | |------|----------| | `tonic_epochs_.csv` | The epoch windows defined for one recording site | | `tonic_.h5` | Each epoch's mean z-score and mean ΔF/F | Both files are written per recording site, and only for sites that have at least one epoch window. Saving with a site's windows cleared deletes that site's pair. **`tonic_epochs_.csv`** has a header row and one row per epoch, with columns `label`, `start` and `end`. `start` and `end` are in seconds of absolute session time, on the same timebase as `timestampNew`. There is no index column. **`tonic_.h5`** holds a DataFrame under the key `df`, indexed by the epoch labels from the CSV (index name `epoch`), with columns `mean_zscore` and `mean_dff` — the mean of `z_score_.hdf5` and of `dff_.hdf5` over that epoch's window. Windows are clamped to the recording's timespan and NaN samples are excluded from the mean. The means are computed at save time from the traces then on disk, and the differences between epochs shown in the visualization are derived at view time from a selectable baseline epoch — only the absolute means are stored. See [Measure tonic signal levels across a session](../how-to/tonic-analysis.md). --- ## Step 4: Compute the PSTH | File | Contents | |------|----------| | `__.h5` | The PSTH: one column per trial, plus summary columns | | `__baselineUncorrected_.h5` | The same PSTH before baseline correction | | `peak_AUC___.h5` and `.csv` | Peak amplitude and area under the curve | | `freqAndAmp_.h5` and `.csv` | Transient frequency and mean amplitude, one row | | `transientsOccurrences_.csv` | One row per detected transient | | `transient_outputs_.hdf5` | The trace and peak indices behind the transient results | | `transients_.hdf5` | The transient times as an event train, **Use Transients as Events?** runs only | | `binned_metrics_.h5` and `.csv` | One row per fixed-width time bin, when **Compute Binned Metrics?** is enabled | | `binned_covariates_.h5` and `.csv` | One row per time bin, holding the behavioral covariate means | | `covariate_correlations_.h5` and `.csv` | One row per metric–covariate pair | | `cross_correlation_output/corr____.h5` | Cross-correlation between two recording sites | | `psth_significance_output/significance_.h5` and `.csv` | Which stretches of the PSTH window are significant | **The PSTH files** hold a `float32` DataFrame under the key `df`, with one row per time point in the peri-event window. The columns, in order: one column per trial, labeled with that trial's event timestamp as a string; then, when trial binning is enabled, a `bin_(