DLRM run failed in torchrec+lightning
model: recipes/torchrecipes/rec at main · facebookresearch/recipes · GitHub error: dlrm_main/0 [0]:[rank0]: Traceback (most recent call last): dlrm_main/0 [0]:[rank0]: File...
View ArticleFine tuning using LLAMA models
Hello, My code was working with the T5 model for finetuning # train.py import os import torch import datasets from transformers import T5ForConditionalGeneration, T5Tokenizer import lightning as L...
View ArticleSave torchmetrics plots after logging them in LightningModule
Hello, I am using a LightningModule and a Trainer and I’m using multiple Metrics from torchmetrics, some are native metrics to the library and some are customized Metrics objects. I’m only interested...
View ArticleGo pass the sanity check but get CUDA OUT OF MEMORY when in validation loop
Hi, when I run the train code. It pass the sanity check and use about 15GB/24GB memory. But when the code went to validation loop, I got CUDA OUT OF MEMORY error (it was fine in train loop. my...
View ArticleLightningModule.train_dataloader()
How do the hooks for the LightningModule interact with the hooks for the LightningDataModule? Does one overwrite the other? Previously, I was able to call the LightningDataModule.train_dataloader()...
View ArticleSkip instances during training
Hi, I am using the LightningModule to train a neural network across many instances/GPUs, however the data is imbalanced ( I cannot change this ), so I want to skip over some instances during training...
View ArticleDo I need to detach when using self.logger.experiment.add_scalars?
I am aware that when we use self.log("train_loss",loss) for instance, the loss tensor is automatically detached to avoid CPU RAM leak. However, if I am logging something else through the method...
View ArticleHow to interactively run inference with a model in jupyter notebook created...
example: RAD-MMM/tts_main.py at main · NVIDIA/RAD-MMM · GitHub 1 post - 1 participant Read full topic
View ArticleSave and restore persisted DataLoader states from checkpoint
Hi! I am working on a project to save and restore persisted DataLoader states from checkpoint, especially working with vanilla Pytorch DataLoader Can you provide suggestions on how to implement that?...
View Articleconfusions about load_from_checkpoint() and save_hyperparameters()
according to Saving and loading checkpoints (basic) — PyTorch Lightning 2.1.3 documentation, There is a model like this: class Encoder(L.LightningModule): ... class Decoder(L.LightningModule): ......
View ArticleHow can I remove metric parameters from model?
Hi, I meet a problem that lightning will save my metric parameters and make pytorch cannot load weights directly, how can I exlude it? Below is my code and class IAT_enhancement(L.LightningModule):...
View ArticleRuntimeError: one of the variables needed for gradient computation has been...
My first forward pass went on smoothly but then i encounter this runtime error Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: one of the...
View ArticleMixed precision training (how to appropriately scale the manual gradient...
I’m working with mixed precision training. My loss has conceptually two components: loss1 and loss2. I call self.manual_backward(loss1,retain_graph=True). This fills gradients to all params. For...
View ArticleMetrics not logged properly in PyTorch Lightning
The feature of logging is not working fine. It is giving following logs on console → v_num:z3_3 val_loss:3.105 val_kappa:0.34 val_accuracy:0.295 train_loss:2.436 train_kappa: nan train_accuracy:0.0...
View ArticleClik here to view.
Question about recover nested model from checkpoint
I have a Nested model class MovieScoreTask(pl.LightningModule): def __init__(self, base_model:nn.Module, learning_rate:float): super().__init__() self.save_hyperparameters() # self.example_input_array...
View ArticleValueError: too many values to unpack (expected 3)
For studying purposes, I am trying to create a simple fine-tuning example using t5 and lighting: import pandas as pd df = pd.DataFrame({ "text": ["O Brasil é um país localizado na América do Sul.", "A...
View ArticleLogging one value per epoch?
Reading the documentation and following the examples, there doesn’t seem to be a way to log just one value per epoch. This is insane, because when you’re trying to figure out a model architecture,...
View ArticleERROR:root:Attempting to deserialize object on a CUDA device but...
Dear I trained a model that came from huggingface and the training works and saving the checkpoint. After when I try to load the model on a pc withouth CUDA I obtain the error: ERROR:root:Attempting...
View ArticleLoad checkpoint with dynamically created model
Hi, In the Lightingmodule docs, the setup hook is described as a possibility to dynamically build a model (instead of initiating in __init__). See the example here. However, when I load a...
View ArticleWhere should I load the model checkpoint when using configure_model?
When i load the model checkpoint in configure_model, the following error occurs. It seems to create an empty model, where should I load the model checkpoint? size mismatch for...
View Article
![ONEW – KAKERA -Unmei no Piece- – EP [iTunes Plus M4A]](http://is1-ssl.mzstatic.com/image/thumb/Music221/v4/d0/fb/86/d0fb86a5-ba60-4bea-41fb-7f54c125f518/VEATP-45908.jpg/296x296bb.webp)
/pic9814888.png)