15-06-v2

README.md

@@ -147,3 +147,205 @@ If you find this work useful for your research, please cite our paper:
 For any questions, please contact the following authors via email or feel free to open an issue 😊
 - Sunkyung Lee sk1027@skku.edu
 - Minjin Choi zxcvxd@skku.edu
+
+# GLEN Model for The Vault Dataset
+
+This repository contains the implementation of the GLEN (Generative Language ENcoder) model for document retrieval and query processing on The Vault dataset.
+
+## Table of Contents
+- [Prerequisites](#prerequisites)
+- [Environment Setup](#environment-setup)
+- [Data Preparation](#data-preparation)
+- [Quick Testing](#quick-testing)
+- [Full Training](#full-training)
+- [Model Evaluation](#model-evaluation)
+- [Troubleshooting](#troubleshooting)
+
+## Prerequisites
+
+- Python 3.8 or higher
+- CUDA-capable GPU (recommended) or CPU
+- Git
+- pip (Python package manager)
+
+## Environment Setup
+
+1. Clone the repository:
+```bash
+git clone <repository-url>
+cd GLEN-model
+```
+
+2. Create and activate a virtual environment:
+```bash
+# Windows
+python -m venv .env
+.env\Scripts\activate
+
+# Linux/Mac
+python -m venv .env
+source .env/bin/activate
+```
+
+3. Install required packages:
+```bash
+pip install -r requirements.txt
+```
+
+4. Create necessary directories:
+```bash
+mkdir -p logs/test_glen_vault
+mkdir -p data/the_vault
+```
+
+## Data Preparation
+
+1. Place your dataset in the `the_vault_dataset` directory:
+```
+the_vault_dataset/
+├── DOC_VAULT_train.tsv
+├── GTQ_VAULT_train.tsv
+└── GTQ_VAULT_dev.tsv
+```
+
+2. Run data preprocessing:
+```bash
+python scripts/preprocess_vault_dataset.py \
+    --input_dir the_vault_dataset/ \
+    --output_dir data/the_vault/ \
+    --sample_size 1000 \
+    --create_test_set
+```
+
+## Quick Testing
+
+To test the model with a small dataset (1000 samples):
+
+1. Run the test script:
+```bash
+bash scripts/test_small_training.sh
+```
+
+This script will:
+- Preprocess a small subset of data
+- Train Phase 1 (Document ID Assignment)
+- Train Phase 2 (Ranking-based Refinement)
+- Generate document IDs
+- Run query inference
+
+Expected output directories:
+```
+logs/test_glen_vault/
+├── GLEN_P1_test/           # Phase 1 model
+├── GLEN_P2_test/           # Phase 2 model
+└── GLEN_P2_test_docids.tsv # Generated document IDs
+```
+
+## Full Training
+
+To train the model on the complete dataset:
+
+1. Run the full training script:
+```bash
+bash scripts/train_full_vault.sh
+```
+
+This script will:
+- Use the entire dataset
+- Train both phases with full parameters
+- Generate document IDs for all documents
+- Run comprehensive query inference
+
+Expected output directories:
+```
+logs/glen_vault/
+├── GLEN_P1/           # Phase 1 model
+├── GLEN_P2/           # Phase 2 model
+└── GLEN_P2_docids.tsv # Generated document IDs
+```
+
+## Model Evaluation
+
+After training, you can evaluate the model:
+
+1. For test results:
+```bash
+python examples/glen_phase2/evaluate_glen.py \
+    --model_name_or_path logs/glen_vault/GLEN_P2 \
+    --infer_dir logs/glen_vault/GLEN_P2 \
+    --dataset_name the_vault \
+    --docid_file_name GLEN_P2_docids \
+    --per_device_eval_batch_size 1 \
+    --q_max_len 32 \
+    --num_return_sequences 5 \
+    --logs_dir logs/glen_vault
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **CUDA Out of Memory**:
+   - Reduce batch sizes in the training scripts
+   - Enable gradient accumulation
+   - Use smaller model (e.g., t5-small instead of t5-base)
+
+2. **CPU Training is Slow**:
+   - Reduce dataset size for testing
+   - Increase gradient accumulation steps
+   - Use smaller batch sizes
+
+3. **Missing Files**:
+   - Ensure all required directories exist
+   - Check file permissions
+   - Verify data preprocessing completed successfully
+
+### Resource Requirements
+
+Minimum recommended specifications:
+- CPU: 8 cores
+- RAM: 16GB
+- GPU: 8GB VRAM (for full training)
+- Storage: 10GB free space
+
+### Performance Tips
+
+1. For CPU-only training:
+   - Use smaller batch sizes (1-2)
+   - Increase gradient accumulation steps
+   - Disable dataloader workers
+   - Use FP16 precision
+
+2. For GPU training:
+   - Adjust batch sizes based on GPU memory
+   - Enable dataloader workers
+   - Use mixed precision training
+
+## Directory Structure
+
+```
+GLEN-model/
+├── data/
+│   └── the_vault/           # Processed dataset
+├── examples/
+│   ├── glen_phase1/        # Phase 1 implementation
+│   └── glen_phase2/        # Phase 2 implementation
+├── logs/
+│   ├── test_glen_vault/    # Test run outputs
+│   └── glen_vault/         # Full training outputs
+├── scripts/
+│   ├── preprocess_vault_dataset.py
+│   ├── test_small_training.sh
+│   └── train_full_vault.sh
+├── .env/                   # Virtual environment
+├── requirements.txt        # Python dependencies
+└── README.md              # This file
+```
+
+## License
+
+[Add your license information here]
+
+## Citation
+
+[Add citation information here]

logs/test_glen_vault/GLEN_P1_test/checkpoint-6/config.json

@@ -0,0 +1,31 @@
+{
+  "Rdrop": 0.15,
+  "architectures": [
+    "T5ForConditionalGeneration_GLEN"
+  ],
+  "d_ff": 3072,
+  "d_kv": 64,
+  "d_model": 768,
+  "decode_vocab_size": 32128,
+  "decoder_start_token_id": 0,
+  "dropout_rate": 0.1,
+  "eos_token_id": 1,
+  "eval_batch_size": 4,
+  "initializer_factor": 1.0,
+  "input_dropout": 1,
+  "is_encoder_decoder": true,
+  "layer_norm_epsilon": 1e-06,
+  "model_type": "t5",
+  "n_positions": 512,
+  "num_decoder_layers": 12,
+  "num_heads": 12,
+  "num_layers": 12,
+  "output_past": true,
+  "pad_token_id": 0,
+  "relative_attention_num_buckets": 32,
+  "tie_decode_embedding": true,
+  "torch_dtype": "float32",
+  "train_batch_size": 8,
+  "transformers_version": "4.52.4",
+  "vocab_size": 32128
+}

logs/test_glen_vault/GLEN_P1_test/checkpoint-6/trainer_state.json

@@ -0,0 +1,33 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0,
+  "eval_steps": 6,
+  "global_step": 6,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [],
+  "logging_steps": 100,
+  "max_steps": 6,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 6,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 0.0,
+  "train_batch_size": 8,
+  "trial_name": null,
+  "trial_params": null
+}

logs/test_glen_vault/GLEN_P1_test/config.json

@@ -10,7 +10,7 @@
   "decoder_start_token_id": 0,
   "dropout_rate": 0.1,
   "eos_token_id": 1,
-  "eval_batch_size": 1,
+  "eval_batch_size": 4,
   "initializer_factor": 1.0,
   "input_dropout": 1,
   "is_encoder_decoder": true,
@@ -25,7 +25,7 @@
   "relative_attention_num_buckets": 32,
   "tie_decode_embedding": true,
   "torch_dtype": "float32",
-  "train_batch_size": 2,
+  "train_batch_size": 8,
   "transformers_version": "4.52.4",
   "vocab_size": 32128
 }

logs/test_glen_vault/GLEN_P1_test/model_args.json

@@ -24,7 +24,7 @@
     "infer_ckpt": "",
     "infer_dir": "",
     "logs_dir": "logs",
-    "docid_file_name": "",
+    "docid_file_name": "logs/test_glen_vault/GLEN_P1_test\\GLENP1Model_len_128_the_vault.tsv",
     "verbose_valid_query": 1,
     "freeze_encoder": false,
     "freeze_embeds": false,

logs/test_glen_vault/GLEN_P2_test/checkpoint-7/config.json

@@ -12,7 +12,7 @@
   "dense_act_fn": "relu",
   "dropout_rate": 0.1,
   "eos_token_id": 1,
-  "eval_batch_size": 1,
+  "eval_batch_size": 4,
   "feed_forward_proj": "relu",
   "id2label": {
     "0": "LABEL_0"
@@ -36,7 +36,7 @@
   "relative_attention_num_buckets": 32,
   "tie_decode_embedding": true,
   "torch_dtype": "float32",
-  "train_batch_size": 2,
+  "train_batch_size": 8,
   "transformers_version": "4.52.4",
   "use_cache": true,
   "vocab_size": 32128

logs/test_glen_vault/GLEN_P2_test/checkpoint-7/model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ca23eacbe2031cec8dd8c5081e9ca6a8e598df1db217aef9a10c5bb38592a56e
+oid sha256:d4c3a8544cae4f0ca7d58d039d2d5943cb33c4ef70b01a6dacc2780718f08454
 size 891644712

logs/test_glen_vault/GLEN_P2_test/checkpoint-7/trainer_state.json

@@ -27,7 +27,7 @@
     }
   },
   "total_flos": 0.0,
-  "train_batch_size": 2,
+  "train_batch_size": 4,
   "trial_name": null,
   "trial_params": null
 }

logs/test_glen_vault/GLEN_P2_test/data_args.json

@@ -1,7 +1,7 @@
 {
     "dataset_name": "the_vault",
     "encode_train_qry": false,
-    "test100": 1,
+    "test100": 0,
     "query_type": "gtq_doc_aug_qg",
     "small_set": 0,
     "aug_query": true,

logs/test_glen_vault/GLEN_P2_test/model_args.json

@@ -24,7 +24,7 @@
     "infer_ckpt": "",
     "infer_dir": "",
     "logs_dir": "logs",
-    "docid_file_name": "",
+    "docid_file_name": "logs/test_glen_vault/GLEN_P2_test\\GLENP2Model_len_128_the_vault.tsv",
     "softmax_temperature": 1.0,
     "num_multi_vectors": 3,
     "untie_encoder": false,
@@ -34,6 +34,7 @@
     "do_docid_temperature_annealing": true,
     "docid_temperature": 1.0,
     "docid_temperature_min": 1e-05,
+    "max_output_length": 4,
     "special_token_ids": [
         2,
         32099,

scripts/preprocess_vault_dataset.py

@@ -126,6 +126,8 @@ def main():
                         help='Include code comments in descriptions')
     parser.add_argument('--max_samples', type=int, default=None,
                         help='Maximum number of samples to process (for testing)')
+    parser.add_argument('--create_test_set', action='store_true',
+                        help='Create test set for evaluation')
     
     args = parser.parse_args()
     
@@ -187,9 +189,9 @@ def main():
         # Create query-document pairs for evaluation data
         elif split in ['validate', 'test']:
             pairs = create_query_document_pairs(processed_samples)
-            eval_split = 'dev' if split == 'validate' else 'test'
+            # Always use 'dev' for evaluation to match GLEN's expectations
             gtq_df = pd.DataFrame(pairs)
-            gtq_file = os.path.join(args.output_dir, f"GTQ_VAULT_{eval_split}.tsv")
+            gtq_file = os.path.join(args.output_dir, "GTQ_VAULT_dev.tsv")
             gtq_df.to_csv(gtq_file, sep='\t', index=False, encoding='utf-8')
             print(f"Saved evaluation query-document pairs to {gtq_file}")
         
@@ -198,22 +200,9 @@ def main():
         
         # Create separate ID file for each split
         if split == 'train_small':
-            id_file = os.path.join(args.output_dir, f"ID_VAULT_t5_bm25_truncate_3.tsv")
+            id_file = os.path.join(args.output_dir, "ID_VAULT_t5_bm25_truncate_3.tsv")
             id_df.to_csv(id_file, sep='\t', index=False, encoding='utf-8')
-            print(f"Created document IDs in {id_file}")
-        else:
-            # For validation and test, create separate ID files if needed
-            eval_split = 'dev' if split == 'validate' else 'test'
-            id_file = os.path.join(args.output_dir, f"ID_VAULT_{eval_split}_t5_bm25_truncate_3.tsv")
-            id_df.to_csv(id_file, sep='\t', index=False, encoding='utf-8')
-            print(f"Created document IDs in {id_file}")
-    
-    print("Preprocessing completed!")
-    print(f"Output files saved in: {args.output_dir}")
-    print("\nGenerated files:")
-    print("- DOC_VAULT_*.tsv: Document content files")
-    print("- GTQ_VAULT_*.tsv: Query-document pairs for training/evaluation")
-    print("- ID_VAULT_*.tsv: Document ID mappings")
+            print(f"Saved document IDs to {id_file}")
 
 if __name__ == "__main__":
     main() 

scripts/test_small_training.ps1

@@ -5,11 +5,30 @@ Write-Host "Testing GLEN with small Vault dataset"
 Write-Host "==========================================="
 
 # Set memory monitoring parameters
-$GPU_MEMORY_THRESHOLD = 0.8
-$GPU_CHECK_INTERVAL = 10
+$GPU_MEMORY_THRESHOLD = 0.85
+$GPU_CHECK_INTERVAL = 50
+
+Write-Host "GPU Memory Protection enabled:"
+Write-Host "- Memory threshold: ${GPU_MEMORY_THRESHOLD} (85%)"  
+Write-Host "- Check interval: ${GPU_CHECK_INTERVAL} steps"
+Write-Host ""
+
+# Ensure data preprocessing is done
+Write-Host "Checking data preprocessing..."
+if (-not (Test-Path "data/the_vault/DOC_VAULT_train.tsv")) {
+    Write-Host "Running data preprocessing..."
+    python scripts/preprocess_vault_dataset.py --input_dir the_vault_dataset/ --output_dir data/the_vault/ --sample_size 1000
+    if ($LASTEXITCODE -ne 0) {
+        Write-Error "Data preprocessing failed!"
+        exit 1
+    }
+} else {
+    Write-Host "Data already preprocessed."
+}
 
 # Test Phase 1 Training
-Write-Host "Starting Phase 1 training test..."
+Write-Host ""
+Write-Host "=== Phase 1 Training (Document ID Assignment) ==="
 $env:CUDA_VISIBLE_DEVICES = "0"
 
 try {
@@ -17,9 +36,9 @@ try {
         --output_dir logs/test_glen_vault/GLEN_P1_test `
         --model_name_or_path t5-base `
         --query_type gtq_doc `
-        --per_device_train_batch_size 2 `
-        --per_device_eval_batch_size 1 `
-        --gradient_accumulation_steps 4 `
+        --per_device_train_batch_size 8 `
+        --per_device_eval_batch_size 4 `
+        --gradient_accumulation_steps 2 `
         --dropout_rate 0.1 `
         --Rdrop 0.15 `
         --aug_query True `
@@ -57,25 +76,42 @@ try {
     exit 1
 }
 
-Write-Host "Phase 1 training completed successfully!"
+Write-Host "✅ Phase 1 training completed successfully!"
 
 # Check if Phase 1 checkpoint exists
 $PHASE1_CKPT = "logs/test_glen_vault/GLEN_P1_test"
 if (-not (Test-Path $PHASE1_CKPT)) {
-    Write-Error "Phase 1 checkpoint not found at $PHASE1_CKPT"
+    Write-Error "❌ Phase 1 checkpoint not found at $PHASE1_CKPT"
     exit 1
 }
 
-Write-Host "Starting Phase 2 training test..."
+# Check for model files
+$model_files = @("pytorch_model.bin", "model.safetensors")
+$found_model = $false
+foreach ($file in $model_files) {
+    if (Test-Path "$PHASE1_CKPT/$file") {
+        $found_model = $true
+        Write-Host "📁 Found Phase 1 model: $file"
+        break
+    }
+}
+
+if (-not $found_model) {
+    Write-Error "❌ No model files found in Phase 1 checkpoint"
+    exit 1
+}
+
+Write-Host ""
+Write-Host "=== Phase 2 Training (Ranking-based Refinement) ==="
 
 # Test Phase 2 Training
 try {
     python examples/glen_phase2/train_glen.py `
         --output_dir logs/test_glen_vault/GLEN_P2_test `
         --model_name_or_path $PHASE1_CKPT `
-        --per_device_train_batch_size 2 `
-        --per_device_eval_batch_size 1 `
-        --gradient_accumulation_steps 8 `
+        --per_device_train_batch_size 4 `
+        --per_device_eval_batch_size 2 `
+        --gradient_accumulation_steps 4 `
         --dropout_rate 0.1 `
         --warmup_ratio 0.1 `
         --id_class t5_bm25_truncate_3 `
@@ -109,22 +145,52 @@ try {
     exit 1
 }
 
-Write-Host "Phase 2 training completed successfully!"
+Write-Host "✅ Phase 2 training completed successfully!"
 
-# Test Document ID Generation
-Write-Host "Testing document ID generation..."
+# Validate Phase 2 checkpoint
 $PHASE2_CKPT = "logs/test_glen_vault/GLEN_P2_test"
+if (-not (Test-Path $PHASE2_CKPT)) {
+    Write-Error "❌ Phase 2 checkpoint not found at $PHASE2_CKPT"
+    exit 1
+}
+
+# Check for checkpoint subdirectories or model files
+$checkpoint_dirs = Get-ChildItem -Path $PHASE2_CKPT -Directory -Name "checkpoint-*" | Sort-Object {[int]($_.Split('-')[1])} | Select-Object -Last 1
+if ($checkpoint_dirs) {
+    Write-Host "📁 Found Phase 2 checkpoint: $checkpoint_dirs"
+    $checkpoint_path = "$PHASE2_CKPT/$checkpoint_dirs"
+    if (-not (Test-Path "$checkpoint_path/model.safetensors") -and -not (Test-Path "$checkpoint_path/pytorch_model.bin")) {
+        Write-Error "❌ No model files in checkpoint directory"
+        exit 1
+    }
+} else {
+    # Check for model files in root
+    $found_model = $false
+    foreach ($file in $model_files) {
+        if (Test-Path "$PHASE2_CKPT/$file") {
+            $found_model = $true
+            Write-Host "📁 Found Phase 2 model: $file"
+            break
+        }
+    }
+    if (-not $found_model) {
+        Write-Error "❌ No model files found in Phase 2 checkpoint"
+        exit 1
+    }
+}
+
+Write-Host ""
+Write-Host "=== Document ID Generation ==="
 
 try {
     python examples/glen_phase2/makeid_glen.py `
         --model_name_or_path $PHASE2_CKPT `
         --infer_dir $PHASE2_CKPT `
         --dataset_name the_vault `
-        --id_class t5_bm25_truncate_3 `
-        --p_max_len 128 `
-        --num_return_sequences 5 `
-        --logs_dir logs/test_glen_vault `
-        --test100 1
+        --docid_file_name GLEN_P2_test_docids `
+        --per_device_eval_batch_size 4 `
+        --max_input_length 128 `
+        --num_return_sequences 10
 
     if ($LASTEXITCODE -ne 0) {
         throw "Document ID generation failed!"
@@ -134,21 +200,29 @@ try {
     exit 1
 }
 
-Write-Host "Document ID generation completed successfully!"
+# Validate docid file was created
+$docid_file = "logs/test_glen_vault/GLEN_P2_test_docids.tsv"
+if (-not (Test-Path $docid_file)) {
+    Write-Error "❌ Document ID file not created: $docid_file"
+    exit 1
+}
 
-# Test Query Inference  
-Write-Host "Testing query inference..."
+$line_count = (Get-Content $docid_file).Count
+Write-Host "✅ Document ID generation completed! Generated $line_count document IDs"
+
+Write-Host ""
+Write-Host "=== Query Inference ==="
 
 try {
     python examples/glen_phase2/evaluate_glen.py `
         --model_name_or_path $PHASE2_CKPT `
         --infer_dir $PHASE2_CKPT `
         --dataset_name the_vault `
-        --id_class t5_bm25_truncate_3 `
+        --docid_file_name GLEN_P2_test_docids `
+        --per_device_eval_batch_size 4 `
         --q_max_len 32 `
         --num_return_sequences 5 `
-        --logs_dir logs/test_glen_vault `
-        --test100 1
+        --logs_dir logs/test_glen_vault
 
     if ($LASTEXITCODE -ne 0) {
         throw "Query inference failed!"
@@ -158,13 +232,27 @@ try {
     exit 1
 }
 
+Write-Host "✅ Query inference completed successfully!"
+
+Write-Host ""
 Write-Host "==========================================="
-Write-Host "All tests completed successfully!"
+Write-Host "🎉 ALL TESTS COMPLETED SUCCESSFULLY! 🎉"
 Write-Host "==========================================="
-Write-Host "Training logs and results saved in: logs/test_glen_vault/"
 Write-Host ""
-Write-Host "GPU Memory Monitoring was active with:"
-Write-Host "- Memory threshold: $GPU_MEMORY_THRESHOLD (80%)"  
-Write-Host "- Check interval: $GPU_CHECK_INTERVAL steps"
+Write-Host "📊 Summary:"
+Write-Host "  ✅ Phase 1 Training (Document ID Assignment)"
+Write-Host "  ✅ Phase 2 Training (Ranking-based Refinement)"  
+Write-Host "  ✅ Document ID Generation ($line_count IDs)"
+Write-Host "  ✅ Query Inference & Evaluation"
+Write-Host ""
+Write-Host "📁 Results saved in: logs/test_glen_vault/"
+Write-Host "📁 Document IDs: $docid_file"
+Write-Host ""
+Write-Host "🛡️ Memory Protection Summary:"
+Write-Host "  - GPU memory threshold: ${GPU_MEMORY_THRESHOLD} (85%)"
+Write-Host "  - Check interval: ${GPU_CHECK_INTERVAL} steps"
+Write-Host "  - FP16 training enabled"
+Write-Host "  - Optimized batch sizes used"
 Write-Host ""
-Write-Host "The system is ready for full training on The Vault dataset!" 
+Write-Host "🚀 The system is ready for full training on The Vault dataset!"
+Write-Host "   Use scripts/train_full_vault.ps1 for production training." 

scripts/test_small_training.sh

@@ -1,23 +1,56 @@
 #!/bin/bash
 
 echo "==========================================="
-echo "Testing GLEN with small Vault dataset"
+echo "Testing GLEN on The Vault dataset (Small)"
 echo "==========================================="
 
 # Set memory monitoring parameters
-GPU_MEMORY_THRESHOLD=0.8
-GPU_CHECK_INTERVAL=10
+GPU_MEMORY_THRESHOLD=0.85
+GPU_CHECK_INTERVAL=50
+
+echo "Resource Protection enabled:"
+echo "- Memory threshold: ${GPU_MEMORY_THRESHOLD} (85%)"  
+echo "- Check interval: ${GPU_CHECK_INTERVAL} steps"
+echo ""
+
+# Ensure data preprocessing is done
+echo "Checking data preprocessing..."
+if [ ! -f "data/the_vault/DOC_VAULT_train.tsv" ] || [ ! -f "data/the_vault/GTQ_VAULT_dev.tsv" ]; then
+    echo "Running data preprocessing..."
+    python scripts/preprocess_vault_dataset.py --input_dir the_vault_dataset/ --output_dir data/the_vault/ --sample_size 1000 --create_test_set
+    if [ $? -ne 0 ]; then
+        echo "Error: Data preprocessing failed!"
+        exit 1
+    fi
+else
+    echo "Data already preprocessed."
+fi
+
+# Phase 1 Training
+echo ""
+echo "=== Phase 1 Training (Document ID Assignment) ==="
+
+# Check if CUDA is available
+if command -v nvidia-smi &> /dev/null; then
+    export CUDA_VISIBLE_DEVICES="0"
+    echo "Using GPU for training"
+    BATCH_SIZE=8
+    EVAL_BATCH_SIZE=4
+    ACCUM_STEPS=2
+else
+    echo "No GPU detected, using CPU with reduced batch sizes"
+    BATCH_SIZE=2
+    EVAL_BATCH_SIZE=1
+    ACCUM_STEPS=8
+fi
 
-# Test Phase 1 Training
-echo "Starting Phase 1 training test..."
-CUDA_VISIBLE_DEVICES=0 \
 python examples/glen_phase1/train_glen.py \
     --output_dir logs/test_glen_vault/GLEN_P1_test \
     --model_name_or_path t5-base \
     --query_type gtq_doc \
-    --per_device_train_batch_size 2 \
-    --per_device_eval_batch_size 1 \
-    --gradient_accumulation_steps 4 \
+    --per_device_train_batch_size $BATCH_SIZE \
+    --per_device_eval_batch_size $EVAL_BATCH_SIZE \
+    --gradient_accumulation_steps $ACCUM_STEPS \
     --dropout_rate 0.1 \
     --Rdrop 0.15 \
     --aug_query True \
@@ -34,47 +67,76 @@ python examples/glen_phase1/train_glen.py \
     --decoder_input doc_rep \
     --max_output_length 5 \
     --num_return_sequences 5 \
-    --logging_steps 10 \
+    --logging_steps 100 \
     --overwrite_output_dir \
-    --wandb_tag test_glen_vault_p1 \
-    --do_eval False \
+    --wandb_tag glen_vault_test_p1 \
+    --do_eval True \
     --num_train_epochs 1 \
-    --save_steps 50 \
+    --save_steps 1000 \
     --save_strategy steps \
-    --evaluation_strategy no \
+    --evaluation_strategy steps \
+    --eval_steps 1000 \
     --seed 42 \
-    --gpu_memory_threshold ${GPU_MEMORY_THRESHOLD} \
-    --gpu_check_interval ${GPU_CHECK_INTERVAL} \
-    --fp16 True
+    --gpu_memory_threshold $GPU_MEMORY_THRESHOLD \
+    --gpu_check_interval $GPU_CHECK_INTERVAL \
+    --fp16 True \
+    --dataloader_num_workers 0 \
+    --dataloader_pin_memory False
 
 if [ $? -ne 0 ]; then
-    echo "Phase 1 training failed!"
+    echo "Error: Phase 1 training failed!"
     exit 1
 fi
 
-echo "Phase 1 training completed successfully!"
+echo "✅ Phase 1 training completed successfully!"
 
 # Check if Phase 1 checkpoint exists
 PHASE1_CKPT="logs/test_glen_vault/GLEN_P1_test"
 if [ ! -d "$PHASE1_CKPT" ]; then
-    echo "Phase 1 checkpoint not found at $PHASE1_CKPT"
+    echo "Error: Phase 1 checkpoint not found at $PHASE1_CKPT"
+    exit 1
+fi
+
+# Check for model files
+model_files=("pytorch_model.bin" "model.safetensors")
+found_model=false
+for file in "${model_files[@]}"; do
+    if [ -f "$PHASE1_CKPT/$file" ]; then
+        found_model=true
+        echo "📁 Found Phase 1 model: $file"
+        break
+    fi
+done
+
+if [ "$found_model" = false ]; then
+    echo "Error: No model files found in Phase 1 checkpoint"
     exit 1
 fi
 
-echo "Starting Phase 2 training test..."
-# Test Phase 2 Training
-CUDA_VISIBLE_DEVICES=0 \
+echo ""
+echo "=== Phase 2 Training (Ranking-based Refinement) ==="
+
+# Adjust batch sizes for Phase 2
+if command -v nvidia-smi &> /dev/null; then
+    BATCH_SIZE=4
+    EVAL_BATCH_SIZE=2
+    ACCUM_STEPS=4
+else
+    BATCH_SIZE=1
+    EVAL_BATCH_SIZE=1
+    ACCUM_STEPS=16
+fi
+
 python examples/glen_phase2/train_glen.py \
     --output_dir logs/test_glen_vault/GLEN_P2_test \
-    --model_name_or_path ${PHASE1_CKPT} \
-    --per_device_train_batch_size 2 \
-    --per_device_eval_batch_size 1 \
-    --gradient_accumulation_steps 8 \
+    --model_name_or_path $PHASE1_CKPT \
+    --per_device_train_batch_size $BATCH_SIZE \
+    --per_device_eval_batch_size $EVAL_BATCH_SIZE \
+    --gradient_accumulation_steps $ACCUM_STEPS \
     --dropout_rate 0.1 \
     --warmup_ratio 0.1 \
     --id_class t5_bm25_truncate_3 \
     --dataset_name the_vault \
-    --test100 1 \
     --tree 1 \
     --q_max_len 32 \
     --p_max_len 128 \
@@ -82,73 +144,136 @@ python examples/glen_phase2/train_glen.py \
     --positive_passage_no_shuffle True \
     --tie_word_embeddings True \
     --num_return_sequences 5 \
-    --logging_steps 10 \
+    --logging_steps 100 \
     --overwrite_output_dir \
-    --wandb_tag test_glen_vault_p2 \
-    --do_eval False \
+    --wandb_tag glen_vault_test_p2 \
+    --do_eval True \
     --num_train_epochs 1 \
-    --save_steps 50 \
+    --save_steps 1000 \
     --save_strategy steps \
-    --evaluation_strategy no \
+    --evaluation_strategy steps \
+    --eval_steps 1000 \
     --seed 42 \
-    --gpu_memory_threshold ${GPU_MEMORY_THRESHOLD} \
-    --gpu_check_interval ${GPU_CHECK_INTERVAL} \
-    --fp16 True
+    --gpu_memory_threshold $GPU_MEMORY_THRESHOLD \
+    --gpu_check_interval $GPU_CHECK_INTERVAL \
+    --fp16 True \
+    --dataloader_num_workers 0 \
+    --dataloader_pin_memory False
 
 if [ $? -ne 0 ]; then
-    echo "Phase 2 training failed!"
+    echo "Error: Phase 2 training failed!"
     exit 1
 fi
 
-echo "Phase 2 training completed successfully!"
+echo "✅ Phase 2 training completed successfully!"
 
-# Test Document ID Generation
-echo "Testing document ID generation..."
+# Validate Phase 2 checkpoint
 PHASE2_CKPT="logs/test_glen_vault/GLEN_P2_test"
+if [ ! -d "$PHASE2_CKPT" ]; then
+    echo "Error: Phase 2 checkpoint not found at $PHASE2_CKPT"
+    exit 1
+fi
+
+# Check for checkpoint subdirectories or model files
+checkpoint_dir=$(find "$PHASE2_CKPT" -maxdepth 1 -type d -name "checkpoint-*" | sort -V | tail -n 1)
+if [ -n "$checkpoint_dir" ]; then
+    echo "📁 Found Phase 2 checkpoint: $(basename $checkpoint_dir)"
+    if [ ! -f "$checkpoint_dir/model.safetensors" ] && [ ! -f "$checkpoint_dir/pytorch_model.bin" ]; then
+        echo "Error: No model files in checkpoint directory"
+        exit 1
+    fi
+else
+    # Check for model files in root
+    found_model=false
+    for file in "${model_files[@]}"; do
+        if [ -f "$PHASE2_CKPT/$file" ]; then
+            found_model=true
+            echo "📁 Found Phase 2 model: $file"
+            break
+        fi
+    done
+    if [ "$found_model" = false ]; then
+        echo "Error: No model files found in Phase 2 checkpoint"
+        exit 1
+    fi
+fi
+
+echo ""
+echo "=== Document ID Generation ==="
 
-CUDA_VISIBLE_DEVICES=0 \
 python examples/glen_phase2/makeid_glen.py \
-    --model_name_or_path ${PHASE2_CKPT} \
-    --infer_dir ${PHASE2_CKPT} \
+    --model_name_or_path $PHASE2_CKPT \
+    --infer_dir $PHASE2_CKPT \
     --dataset_name the_vault \
-    --id_class t5_bm25_truncate_3 \
-    --p_max_len 128 \
-    --num_return_sequences 5 \
-    --logs_dir logs/test_glen_vault \
-    --test100 1
+    --docid_file_name GLEN_P2_test_docids \
+    --per_device_eval_batch_size 1 \
+    --max_input_length 128 \
+    --num_return_sequences 10 \
+    --dataloader_num_workers 0 \
+    --dataloader_pin_memory False
 
 if [ $? -ne 0 ]; then
-    echo "Document ID generation failed!"
+    echo "Error: Document ID generation failed!"
+    exit 1
+fi
+
+# Validate docid file was created
+docid_file="logs/test_glen_vault/GLEN_P2_test_docids.tsv"
+if [ ! -f "$docid_file" ]; then
+    echo "Error: Document ID file not created: $docid_file"
     exit 1
 fi
 
-echo "Document ID generation completed successfully!"
+line_count=$(wc -l < "$docid_file")
+echo "✅ Document ID generation completed! Generated $line_count document IDs"
+
+echo ""
+echo "=== Query Inference ==="
+
+# First, ensure we have test queries
+if [ ! -f "data/the_vault/GTQ_VAULT_dev.tsv" ]; then
+    echo "Error: Test queries file not found. Please run preprocessing with --create_test_set flag"
+    exit 1
+fi
 
-# Test Query Inference  
-echo "Testing query inference..."
-CUDA_VISIBLE_DEVICES=0 \
 python examples/glen_phase2/evaluate_glen.py \
-    --model_name_or_path ${PHASE2_CKPT} \
-    --infer_dir ${PHASE2_CKPT} \
+    --model_name_or_path $PHASE2_CKPT \
+    --infer_dir $PHASE2_CKPT \
     --dataset_name the_vault \
-    --id_class t5_bm25_truncate_3 \
+    --docid_file_name GLEN_P2_test_docids \
+    --per_device_eval_batch_size 1 \
     --q_max_len 32 \
     --num_return_sequences 5 \
     --logs_dir logs/test_glen_vault \
-    --test100 1
+    --test100 1 \
+    --dataloader_num_workers 0 \
+    --dataloader_pin_memory False
 
 if [ $? -ne 0 ]; then
-    echo "Query inference failed!"
+    echo "Error: Query inference failed!"
     exit 1
 fi
 
+echo "✅ Query inference completed successfully!"
+
+echo ""
 echo "==========================================="
-echo "All tests completed successfully!"
+echo "🎉 TESTING COMPLETED SUCCESSFULLY! 🎉"
 echo "==========================================="
-echo "Training logs and results saved in: logs/test_glen_vault/"
 echo ""
-echo "GPU Memory Monitoring was active with:"
-echo "- Memory threshold: ${GPU_MEMORY_THRESHOLD} (80%)"  
-echo "- Check interval: ${GPU_CHECK_INTERVAL} steps"
+echo "📊 Summary:"
+echo "  ✅ Phase 1 Training (Document ID Assignment)"
+echo "  ✅ Phase 2 Training (Ranking-based Refinement)"  
+echo "  ✅ Document ID Generation ($line_count IDs)"
+echo "  ✅ Query Inference & Evaluation"
+echo ""
+echo "📁 Results saved in: logs/test_glen_vault/"
+echo "📁 Document IDs: $docid_file"
+echo ""
+echo "🛡️ Resource Protection Summary:"
+echo "  - Memory threshold: ${GPU_MEMORY_THRESHOLD} (85%)"
+echo "  - Check interval: ${GPU_CHECK_INTERVAL} steps"
+echo "  - FP16 training enabled"
+echo "  - Optimized batch sizes for current hardware"
 echo ""
-echo "The system is ready for full training on The Vault dataset!" 
+echo "🚀 Testing completed! The model is ready for full training." 

scripts/train_full_vault.ps1

@@ -0,0 +1,330 @@
+#!/usr/bin/env pwsh
+
+Write-Host "==========================================="
+Write-Host "GLEN Full Training on The Vault Dataset"
+Write-Host "Processing 34M+ code samples"
+Write-Host "==========================================="
+
+# Production parameters
+$GPU_MEMORY_THRESHOLD = 0.85
+$GPU_CHECK_INTERVAL = 50
+$WANDB_PROJECT = "glen-vault-production"
+
+# Training configuration
+$PHASE1_EPOCHS = 3
+$PHASE2_EPOCHS = 5
+$PHASE1_BATCH_SIZE = 32
+$PHASE2_BATCH_SIZE = 16
+$GRADIENT_ACCUMULATION = 4
+$MAX_INPUT_LENGTH = 256
+$LEARNING_RATE = 5e-5
+
+Write-Host "🔧 Production Configuration:"
+Write-Host "  - Phase 1 epochs: $PHASE1_EPOCHS"
+Write-Host "  - Phase 2 epochs: $PHASE2_EPOCHS"
+Write-Host "  - Phase 1 batch size: $PHASE1_BATCH_SIZE"
+Write-Host "  - Phase 2 batch size: $PHASE2_BATCH_SIZE"
+Write-Host "  - Gradient accumulation: $GRADIENT_ACCUMULATION"
+Write-Host "  - Max input length: $MAX_INPUT_LENGTH"
+Write-Host "  - Learning rate: $LEARNING_RATE"
+Write-Host ""
+
+Write-Host "🛡️ Memory Protection:"
+Write-Host "  - GPU memory threshold: ${GPU_MEMORY_THRESHOLD} (85%)"
+Write-Host "  - Check interval: ${GPU_CHECK_INTERVAL} steps"
+Write-Host "  - FP16 training enabled"
+Write-Host "  - Automatic checkpoint saving on memory limit"
+Write-Host ""
+
+# Check prerequisites
+Write-Host "📋 Checking prerequisites..."
+
+# Check if full dataset exists
+if (-not (Test-Path "the_vault_dataset")) {
+    Write-Error "❌ The Vault dataset not found! Please download and extract to 'the_vault_dataset/'"
+    Write-Host "   Download from: https://github.com/microsoft/CodeXGLUE/tree/main/Code-Code/CodeT5-learning-framework/data"
+    exit 1
+}
+
+# Ensure data preprocessing is done for full dataset
+Write-Host "Checking full dataset preprocessing..."
+if (-not (Test-Path "data/the_vault/DOC_VAULT_train.tsv")) {
+    Write-Host "🔄 Running full dataset preprocessing (this may take 30-60 minutes)..."
+    python scripts/preprocess_vault_dataset.py --input_dir the_vault_dataset/ --output_dir data/the_vault/ --full_dataset
+    if ($LASTEXITCODE -ne 0) {
+        Write-Error "❌ Data preprocessing failed!"
+        exit 1
+    }
+} else {
+    $train_lines = (Get-Content "data/the_vault/DOC_VAULT_train.tsv").Count
+    Write-Host "✅ Full dataset already preprocessed ($train_lines training samples)"
+}
+
+# Check GPU availability
+$gpu_count = 0
+try {
+    $gpu_info = nvidia-smi --query-gpu=name --format=csv,noheader,nounits 2>$null
+    if ($gpu_info) {
+        $gpu_count = ($gpu_info | Measure-Object).Count
+        Write-Host "🖥️  Detected $gpu_count GPU(s): $($gpu_info -join ', ')"
+    }
+} catch {
+    Write-Host "⚠️  No GPU detected, will use CPU (training will be much slower)"
+}
+
+if ($gpu_count -eq 0) {
+    Write-Host "⚠️  Warning: Training on CPU will take days/weeks. Consider using GPU."
+    $response = Read-Host "Continue with CPU training? (y/N)"
+    if ($response -ne "y" -and $response -ne "Y") {
+        Write-Host "Training cancelled."
+        exit 0
+    }
+}
+
+Write-Host ""
+Write-Host "=== Phase 1 Training: Document ID Assignment ==="
+Write-Host "🎯 Learning to assign semantic identifiers to code documents"
+
+$PHASE1_OUTPUT = "logs/glen_vault_production/GLEN_P1"
+$env:CUDA_VISIBLE_DEVICES = "0"
+
+try {
+    python examples/glen_phase1/train_glen.py `
+        --output_dir $PHASE1_OUTPUT `
+        --model_name_or_path t5-base `
+        --query_type gtq_doc `
+        --per_device_train_batch_size $PHASE1_BATCH_SIZE `
+        --per_device_eval_batch_size 8 `
+        --gradient_accumulation_steps $GRADIENT_ACCUMULATION `
+        --learning_rate $LEARNING_RATE `
+        --dropout_rate 0.1 `
+        --Rdrop 0.15 `
+        --aug_query True `
+        --aug_query_type corrupted_query `
+        --input_dropout 1 `
+        --id_class t5_bm25_truncate_3 `
+        --dataset_name the_vault `
+        --tree 1 `
+        --pretrain_decoder True `
+        --max_input_length $MAX_INPUT_LENGTH `
+        --val_check_interval 0.1 `
+        --tie_word_embeddings True `
+        --decoder_input doc_rep `
+        --max_output_length 10 `
+        --num_return_sequences 10 `
+        --logging_steps 100 `
+        --eval_steps 1000 `
+        --save_steps 2000 `
+        --overwrite_output_dir `
+        --wandb_tag "phase1_production" `
+        --project_name $WANDB_PROJECT `
+        --do_eval True `
+        --evaluation_strategy steps `
+        --num_train_epochs $PHASE1_EPOCHS `
+        --save_strategy steps `
+        --save_total_limit 5 `
+        --load_best_model_at_end True `
+        --metric_for_best_model eval_loss `
+        --greater_is_better False `
+        --seed 42 `
+        --gpu_memory_threshold $GPU_MEMORY_THRESHOLD `
+        --gpu_check_interval $GPU_CHECK_INTERVAL `
+        --fp16 True `
+        --dataloader_num_workers 4 `
+        --warmup_ratio 0.1
+
+    if ($LASTEXITCODE -ne 0) {
+        throw "Phase 1 training failed!"
+    }
+} catch {
+    Write-Error "❌ Phase 1 training failed: $_"
+    Write-Host "📁 Check logs in: $PHASE1_OUTPUT"
+    exit 1
+}
+
+Write-Host "✅ Phase 1 training completed successfully!"
+
+# Validate Phase 1 checkpoint
+if (-not (Test-Path $PHASE1_OUTPUT)) {
+    Write-Error "❌ Phase 1 checkpoint not found at $PHASE1_OUTPUT"
+    exit 1
+fi
+
+# Find the best checkpoint
+$best_checkpoint = Get-ChildItem -Path $PHASE1_OUTPUT -Directory -Name "checkpoint-*" | 
+    Sort-Object {[int]($_.Split('-')[1])} | Select-Object -Last 1
+
+if ($best_checkpoint) {
+    Write-Host "📁 Using Phase 1 checkpoint: $best_checkpoint"
+    $PHASE1_CKPT = "$PHASE1_OUTPUT/$best_checkpoint"
+} else {
+    $PHASE1_CKPT = $PHASE1_OUTPUT
+}
+
+Write-Host ""
+Write-Host "=== Phase 2 Training: Ranking-based Refinement ==="
+Write-Host "🎯 Learning to rank and refine document identifiers"
+
+$PHASE2_OUTPUT = "logs/glen_vault_production/GLEN_P2"
+
+try {
+    python examples/glen_phase2/train_glen.py `
+        --output_dir $PHASE2_OUTPUT `
+        --model_name_or_path $PHASE1_CKPT `
+        --per_device_train_batch_size $PHASE2_BATCH_SIZE `
+        --per_device_eval_batch_size 4 `
+        --gradient_accumulation_steps $GRADIENT_ACCUMULATION `
+        --learning_rate $LEARNING_RATE `
+        --dropout_rate 0.1 `
+        --warmup_ratio 0.1 `
+        --id_class t5_bm25_truncate_3 `
+        --dataset_name the_vault `
+        --tree 1 `
+        --q_max_len 64 `
+        --p_max_len $MAX_INPUT_LENGTH `
+        --negative_passage_type self `
+        --positive_passage_no_shuffle True `
+        --tie_word_embeddings True `
+        --num_return_sequences 10 `
+        --logging_steps 100 `
+        --eval_steps 1000 `
+        --save_steps 2000 `
+        --overwrite_output_dir `
+        --wandb_tag "phase2_production" `
+        --project_name $WANDB_PROJECT `
+        --do_eval True `
+        --evaluation_strategy steps `
+        --num_train_epochs $PHASE2_EPOCHS `
+        --save_strategy steps `
+        --save_total_limit 5 `
+        --load_best_model_at_end True `
+        --metric_for_best_model eval_loss `
+        --greater_is_better False `
+        --seed 42 `
+        --gpu_memory_threshold $GPU_MEMORY_THRESHOLD `
+        --gpu_check_interval $GPU_CHECK_INTERVAL `
+        --fp16 True `
+        --dataloader_num_workers 4
+
+    if ($LASTEXITCODE -ne 0) {
+        throw "Phase 2 training failed!"
+    }
+} catch {
+    Write-Error "❌ Phase 2 training failed: $_"
+    Write-Host "📁 Check logs in: $PHASE2_OUTPUT"
+    exit 1
+}
+
+Write-Host "✅ Phase 2 training completed successfully!"
+
+# Validate Phase 2 checkpoint
+if (-not (Test-Path $PHASE2_OUTPUT)) {
+    Write-Error "❌ Phase 2 checkpoint not found at $PHASE2_OUTPUT"
+    exit 1
+}
+
+# Find the best Phase 2 checkpoint
+$best_checkpoint_p2 = Get-ChildItem -Path $PHASE2_OUTPUT -Directory -Name "checkpoint-*" | 
+    Sort-Object {[int]($_.Split('-')[1])} | Select-Object -Last 1
+
+if ($best_checkpoint_p2) {
+    Write-Host "📁 Using Phase 2 checkpoint: $best_checkpoint_p2"
+    $PHASE2_CKPT = "$PHASE2_OUTPUT/$best_checkpoint_p2"
+} else {
+    $PHASE2_CKPT = $PHASE2_OUTPUT
+}
+
+Write-Host ""
+Write-Host "=== Document ID Generation ==="
+Write-Host "🎯 Generating semantic IDs for all documents"
+
+try {
+    python examples/glen_phase2/makeid_glen.py `
+        --model_name_or_path $PHASE2_CKPT `
+        --infer_dir $PHASE2_CKPT `
+        --dataset_name the_vault `
+        --docid_file_name glen_vault_production_docids `
+        --per_device_eval_batch_size 16 `
+        --max_input_length $MAX_INPUT_LENGTH `
+        --num_return_sequences 20
+
+    if ($LASTEXITCODE -ne 0) {
+        throw "Document ID generation failed!"
+    }
+} catch {
+    Write-Error "❌ Document ID generation failed: $_"
+    exit 1
+}
+
+# Validate docid file
+$docid_file = "logs/glen_vault_production/glen_vault_production_docids.tsv"
+if (-not (Test-Path $docid_file)) {
+    Write-Error "❌ Document ID file not created: $docid_file"
+    exit 1
+fi
+
+$total_docs = (Get-Content $docid_file).Count
+Write-Host "✅ Document ID generation completed! Generated $total_docs document IDs"
+
+Write-Host ""
+Write-Host "=== Model Evaluation ==="
+Write-Host "🎯 Evaluating model performance on test set"
+
+try {
+    python examples/glen_phase2/evaluate_glen.py `
+        --model_name_or_path $PHASE2_CKPT `
+        --infer_dir $PHASE2_CKPT `
+        --dataset_name the_vault `
+        --docid_file_name glen_vault_production_docids `
+        --per_device_eval_batch_size 8 `
+        --q_max_len 64 `
+        --num_return_sequences 20 `
+        --logs_dir logs/glen_vault_production
+
+    if ($LASTEXITCODE -ne 0) {
+        throw "Model evaluation failed!"
+    }
+} catch {
+    Write-Error "❌ Model evaluation failed: $_"
+    exit 1
+}
+
+Write-Host "✅ Model evaluation completed successfully!"
+
+# Training completion summary
+$training_time = Get-Date
+Write-Host ""
+Write-Host "==========================================="
+Write-Host "🎉 FULL TRAINING COMPLETED SUCCESSFULLY! 🎉"
+Write-Host "==========================================="
+Write-Host ""
+Write-Host "📊 Training Summary:"
+Write-Host "  ✅ Phase 1: Document ID Assignment ($PHASE1_EPOCHS epochs)"
+Write-Host "  ✅ Phase 2: Ranking Refinement ($PHASE2_EPOCHS epochs)"
+Write-Host "  ✅ Document ID Generation ($total_docs documents)"
+Write-Host "  ✅ Model Evaluation & Metrics"
+Write-Host ""
+Write-Host "📁 Production Model Artifacts:"
+Write-Host "  🏷️  Phase 1 Checkpoint: $PHASE1_CKPT"
+Write-Host "  🏷️  Phase 2 Checkpoint: $PHASE2_CKPT"
+Write-Host "  📄 Document IDs: $docid_file"
+Write-Host "  📊 Evaluation Results: logs/glen_vault_production/"
+Write-Host ""
+Write-Host "🛡️ Memory Protection Summary:"
+Write-Host "  - GPU memory threshold: ${GPU_MEMORY_THRESHOLD} (85%)"
+Write-Host "  - Check interval: ${GPU_CHECK_INTERVAL} steps"
+Write-Host "  - FP16 training enabled throughout"
+Write-Host "  - Automatic checkpoint saving on memory limits"
+Write-Host ""
+Write-Host "📈 Performance Optimizations Used:"
+Write-Host "  - Gradient accumulation: ${GRADIENT_ACCUMULATION}x"
+Write-Host "  - Multi-worker data loading"
+Write-Host "  - Mixed precision training (FP16)"
+Write-Host "  - Memory-efficient batch sizes"
+Write-Host ""
+Write-Host "🚀 Your GLEN model is ready for production use!"
+Write-Host "   - Use the Phase 2 checkpoint for inference"
+Write-Host "   - Document IDs are saved for fast retrieval"
+Write-Host "   - Evaluation metrics are in the logs directory"
+Write-Host ""
+Write-Host "Training completed at: $training_time" 

scripts/train_full_vault.sh

@@ -0,0 +1,247 @@
+#!/bin/bash
+
+echo "==========================================="
+echo "Full Training GLEN on The Vault dataset"
+echo "==========================================="
+
+# Set memory monitoring parameters
+GPU_MEMORY_THRESHOLD=0.85
+GPU_CHECK_INTERVAL=50
+
+echo "GPU Memory Protection enabled:"
+echo "- Memory threshold: ${GPU_MEMORY_THRESHOLD} (85%)"  
+echo "- Check interval: ${GPU_CHECK_INTERVAL} steps"
+echo ""
+
+# Ensure data preprocessing is done
+echo "Checking data preprocessing..."
+if [ ! -f "data/the_vault/DOC_VAULT_train.tsv" ] || [ ! -f "data/the_vault/GTQ_VAULT_dev.tsv" ]; then
+    echo "Running data preprocessing..."
+    python scripts/preprocess_vault_dataset.py --input_dir the_vault_dataset/ --output_dir data/the_vault/ --create_test_set
+    if [ $? -ne 0 ]; then
+        echo "Error: Data preprocessing failed!"
+        exit 1
+    fi
+else
+    echo "Data already preprocessed."
+fi
+
+# Phase 1 Training
+echo ""
+echo "=== Phase 1 Training (Document ID Assignment) ==="
+export CUDA_VISIBLE_DEVICES="0"
+
+python examples/glen_phase1/train_glen.py \
+    --output_dir logs/glen_vault/GLEN_P1 \
+    --model_name_or_path t5-base \
+    --query_type gtq_doc \
+    --per_device_train_batch_size 8 \
+    --per_device_eval_batch_size 4 \
+    --gradient_accumulation_steps 2 \
+    --dropout_rate 0.1 \
+    --Rdrop 0.15 \
+    --aug_query True \
+    --aug_query_type corrupted_query \
+    --input_dropout 1 \
+    --id_class t5_bm25_truncate_3 \
+    --dataset_name the_vault \
+    --test100 1 \
+    --tree 1 \
+    --pretrain_decoder True \
+    --max_input_length 128 \
+    --val_check_interval 1.0 \
+    --tie_word_embeddings True \
+    --decoder_input doc_rep \
+    --max_output_length 5 \
+    --num_return_sequences 5 \
+    --logging_steps 100 \
+    --overwrite_output_dir \
+    --wandb_tag glen_vault_p1 \
+    --do_eval True \
+    --num_train_epochs 3 \
+    --save_steps 1000 \
+    --save_strategy steps \
+    --evaluation_strategy steps \
+    --eval_steps 1000 \
+    --seed 42 \
+    --gpu_memory_threshold $GPU_MEMORY_THRESHOLD \
+    --gpu_check_interval $GPU_CHECK_INTERVAL \
+    --fp16 True
+
+if [ $? -ne 0 ]; then
+    echo "Error: Phase 1 training failed!"
+    exit 1
+fi
+
+echo "✅ Phase 1 training completed successfully!"
+
+# Check if Phase 1 checkpoint exists
+PHASE1_CKPT="logs/glen_vault/GLEN_P1"
+if [ ! -d "$PHASE1_CKPT" ]; then
+    echo "Error: Phase 1 checkpoint not found at $PHASE1_CKPT"
+    exit 1
+fi
+
+# Check for model files
+model_files=("pytorch_model.bin" "model.safetensors")
+found_model=false
+for file in "${model_files[@]}"; do
+    if [ -f "$PHASE1_CKPT/$file" ]; then
+        found_model=true
+        echo "📁 Found Phase 1 model: $file"
+        break
+    fi
+done
+
+if [ "$found_model" = false ]; then
+    echo "Error: No model files found in Phase 1 checkpoint"
+    exit 1
+fi
+
+echo ""
+echo "=== Phase 2 Training (Ranking-based Refinement) ==="
+
+python examples/glen_phase2/train_glen.py \
+    --output_dir logs/glen_vault/GLEN_P2 \
+    --model_name_or_path $PHASE1_CKPT \
+    --per_device_train_batch_size 4 \
+    --per_device_eval_batch_size 2 \
+    --gradient_accumulation_steps 4 \
+    --dropout_rate 0.1 \
+    --warmup_ratio 0.1 \
+    --id_class t5_bm25_truncate_3 \
+    --dataset_name the_vault \
+    --tree 1 \
+    --q_max_len 32 \
+    --p_max_len 128 \
+    --negative_passage_type self \
+    --positive_passage_no_shuffle True \
+    --tie_word_embeddings True \
+    --num_return_sequences 5 \
+    --logging_steps 100 \
+    --overwrite_output_dir \
+    --wandb_tag glen_vault_p2 \
+    --do_eval True \
+    --num_train_epochs 3 \
+    --save_steps 1000 \
+    --save_strategy steps \
+    --evaluation_strategy steps \
+    --eval_steps 1000 \
+    --seed 42 \
+    --gpu_memory_threshold $GPU_MEMORY_THRESHOLD \
+    --gpu_check_interval $GPU_CHECK_INTERVAL \
+    --fp16 True
+
+if [ $? -ne 0 ]; then
+    echo "Error: Phase 2 training failed!"
+    exit 1
+fi
+
+echo "✅ Phase 2 training completed successfully!"
+
+# Validate Phase 2 checkpoint
+PHASE2_CKPT="logs/glen_vault/GLEN_P2"
+if [ ! -d "$PHASE2_CKPT" ]; then
+    echo "Error: Phase 2 checkpoint not found at $PHASE2_CKPT"
+    exit 1
+fi
+
+# Check for checkpoint subdirectories or model files
+checkpoint_dir=$(find "$PHASE2_CKPT" -maxdepth 1 -type d -name "checkpoint-*" | sort -V | tail -n 1)
+if [ -n "$checkpoint_dir" ]; then
+    echo "📁 Found Phase 2 checkpoint: $(basename $checkpoint_dir)"
+    if [ ! -f "$checkpoint_dir/model.safetensors" ] && [ ! -f "$checkpoint_dir/pytorch_model.bin" ]; then
+        echo "Error: No model files in checkpoint directory"
+        exit 1
+    fi
+else
+    # Check for model files in root
+    found_model=false
+    for file in "${model_files[@]}"; do
+        if [ -f "$PHASE2_CKPT/$file" ]; then
+            found_model=true
+            echo "📁 Found Phase 2 model: $file"
+            break
+        fi
+    done
+    if [ "$found_model" = false ]; then
+        echo "Error: No model files found in Phase 2 checkpoint"
+        exit 1
+    fi
+fi
+
+echo ""
+echo "=== Document ID Generation ==="
+
+python examples/glen_phase2/makeid_glen.py \
+    --model_name_or_path $PHASE2_CKPT \
+    --infer_dir $PHASE2_CKPT \
+    --dataset_name the_vault \
+    --docid_file_name GLEN_P2_docids \
+    --per_device_eval_batch_size 4 \
+    --max_input_length 128 \
+    --num_return_sequences 10
+
+if [ $? -ne 0 ]; then
+    echo "Error: Document ID generation failed!"
+    exit 1
+fi
+
+# Validate docid file was created
+docid_file="logs/glen_vault/GLEN_P2_docids.tsv"
+if [ ! -f "$docid_file" ]; then
+    echo "Error: Document ID file not created: $docid_file"
+    exit 1
+fi
+
+line_count=$(wc -l < "$docid_file")
+echo "✅ Document ID generation completed! Generated $line_count document IDs"
+
+echo ""
+echo "=== Query Inference ==="
+
+# First, ensure we have test queries
+if [ ! -f "data/the_vault/GTQ_VAULT_dev.tsv" ]; then
+    echo "Error: Test queries file not found. Please run preprocessing with --create_test_set flag"
+    exit 1
+fi
+
+python examples/glen_phase2/evaluate_glen.py \
+    --model_name_or_path $PHASE2_CKPT \
+    --infer_dir $PHASE2_CKPT \
+    --dataset_name the_vault \
+    --docid_file_name GLEN_P2_docids \
+    --per_device_eval_batch_size 4 \
+    --q_max_len 32 \
+    --num_return_sequences 5 \
+    --logs_dir logs/glen_vault \
+    --test100 1
+
+if [ $? -ne 0 ]; then
+    echo "Error: Query inference failed!"
+    exit 1
+fi
+
+echo "✅ Query inference completed successfully!"
+
+echo ""
+echo "==========================================="
+echo "🎉 FULL TRAINING COMPLETED SUCCESSFULLY! 🎉"
+echo "==========================================="
+echo ""
+echo "📊 Summary:"
+echo "  ✅ Phase 1 Training (Document ID Assignment)"
+echo "  ✅ Phase 2 Training (Ranking-based Refinement)"  
+echo "  ✅ Document ID Generation ($line_count IDs)"
+echo "  ✅ Query Inference & Evaluation"
+echo ""
+echo "📁 Results saved in: logs/glen_vault/"
+echo "📁 Document IDs: $docid_file"
+echo ""
+echo "🛡️ Memory Protection Summary:"
+echo "  - GPU memory threshold: ${GPU_MEMORY_THRESHOLD} (85%)"
+echo "  - Check interval: ${GPU_CHECK_INTERVAL} steps"
+echo "  - FP16 training enabled"
+echo "  - Optimized batch sizes used"
+echo ""
+echo "🚀 Training completed! The model is ready for production use." 

wandb/offline-run-20250615_082823-7mv0nkou/files/requirements.txt

@@ -0,0 +1,64 @@
+accelerate==1.7.0
+aiohappyeyeballs==2.6.1
+aiohttp==3.12.13
+aiosignal==1.3.2
+annotated-types==0.7.0
+attrs==25.3.0
+certifi==2025.4.26
+charset-normalizer==3.4.2
+click==8.2.1
+colorama==0.4.6
+datasets==3.6.0
+dill==0.3.8
+filelock==3.18.0
+frozenlist==1.7.0
+fsspec==2025.3.0
+gitdb==4.0.12
+GitPython==3.1.44
+huggingface-hub==0.33.0
+idna==3.10
+Jinja2==3.1.6
+MarkupSafe==3.0.2
+mpmath==1.3.0
+multidict==6.4.4
+multiprocess==0.70.16
+networkx==3.5
+numpy==2.3.0
+packaging==25.0
+pandas==2.3.0
+pillow==11.2.1
+pip==25.1.1
+platformdirs==4.3.8
+propcache==0.3.2
+protobuf==6.31.1
+psutil==7.0.0
+pyarrow==20.0.0
+pydantic==2.11.7
+pydantic_core==2.33.2
+python-dateutil==2.9.0.post0
+pytz==2025.2
+PyYAML==6.0.2
+regex==2024.11.6
+requests==2.32.4
+safetensors==0.5.3
+sentry-sdk==2.30.0
+setproctitle==1.3.6
+setuptools==80.9.0
+six==1.17.0
+smmap==5.0.2
+sympy==1.14.0
+tevatron==0.0.1
+tokenizers==0.21.1
+torch==2.7.1
+torchaudio==2.7.1
+torchvision==0.22.1
+tqdm==4.67.1
+transformers==4.52.4
+typing_extensions==4.14.0
+typing-inspection==0.4.1
+tzdata==2025.2
+urllib3==2.4.0
+wandb==0.20.1
+xxhash==3.5.0
+yarl==1.20.1
+tevatron==0.0.1

wandb/offline-run-20250615_082823-7mv0nkou/files/wandb-metadata.json

@@ -0,0 +1,111 @@
+{
+  "os":  "Windows-10-10.0.19045-SP0",
+  "python":  "CPython 3.13.5",
+  "startedAt":  "2025-06-15T01:28:24.154471Z",
+  "args":  [
+    "--output_dir",
+    "logs/test_glen_vault/GLEN_P1_test",
+    "--model_name_or_path",
+    "t5-base",
+    "--query_type",
+    "gtq_doc",
+    "--per_device_train_batch_size",
+    "8",
+    "--per_device_eval_batch_size",
+    "4",
+    "--gradient_accumulation_steps",
+    "2",
+    "--dropout_rate",
+    "0.1",
+    "--Rdrop",
+    "0.15",
+    "--aug_query",
+    "True",
+    "--aug_query_type",
+    "corrupted_query",
+    "--input_dropout",
+    "1",
+    "--id_class",
+    "t5_bm25_truncate_3",
+    "--dataset_name",
+    "the_vault",
+    "--test100",
+    "1",
+    "--tree",
+    "1",
+    "--pretrain_decoder",
+    "True",
+    "--max_input_length",
+    "128",
+    "--val_check_interval",
+    "1.0",
+    "--tie_word_embeddings",
+    "True",
+    "--decoder_input",
+    "doc_rep",
+    "--max_output_length",
+    "5",
+    "--num_return_sequences",
+    "5",
+    "--logging_steps",
+    "10",
+    "--overwrite_output_dir",
+    "--wandb_tag",
+    "test_glen_vault_p1",
+    "--do_eval",
+    "False",
+    "--num_train_epochs",
+    "1",
+    "--save_steps",
+    "50",
+    "--save_strategy",
+    "steps",
+    "--evaluation_strategy",
+    "no",
+    "--seed",
+    "42",
+    "--gpu_memory_threshold",
+    "0.85",
+    "--gpu_check_interval",
+    "50",
+    "--fp16",
+    "True"
+  ],
+  "program":  "H:\\Code\\GLEN-model\\examples\\glen_phase1\\train_glen.py",
+  "codePath":  "examples\\glen_phase1\\train_glen.py",
+  "git":  {
+    "remote":  "https://QuanTH02:@huggingface.co/QuanTH02/GLEN-model",
+    "commit":  "6534252bf5ad60b20ba58d7d578a982aabeaacaa"
+  },
+  "root":  "H:\\Code\\GLEN-model",
+  "host":  "FPS-33",
+  "executable":  "H:\\Code\\GLEN-model\\.env\\Scripts\\python.exe",
+  "codePathLocal":  "examples\\glen_phase1\\train_glen.py",
+  "cpu_count":  10,
+  "cpu_count_logical":  16,
+  "gpu":  "NVIDIA GeForce RTX 4060",
+  "gpu_count":  1,
+  "disk":  {
+    "/":  {
+      "total":  "8001561812992",
+      "used":  "3636055900160"
+    }
+  },
+  "memory":  {
+    "total":  "34157170688"
+  },
+  "cpu":  {
+    "count":  10,
+    "countLogical":  16
+  },
+  "gpu_nvidia":  [
+    {
+      "name":  "NVIDIA GeForce RTX 4060",
+      "memoryTotal":  "8585740288",
+      "cudaCores":  3072,
+      "architecture":  "Ada",
+      "uuid":  "GPU-7e0c8403-933a-8533-bde6-f629db871693"
+    }
+  ],
+  "cudaVersion":  "12.8"
+}

wandb/offline-run-20250615_083045-gw7kaqtk/files/requirements.txt

@@ -0,0 +1,64 @@
+accelerate==1.7.0
+aiohappyeyeballs==2.6.1
+aiohttp==3.12.13
+aiosignal==1.3.2
+annotated-types==0.7.0
+attrs==25.3.0
+certifi==2025.4.26
+charset-normalizer==3.4.2
+click==8.2.1
+colorama==0.4.6
+datasets==3.6.0
+dill==0.3.8
+filelock==3.18.0
+frozenlist==1.7.0
+fsspec==2025.3.0
+gitdb==4.0.12
+GitPython==3.1.44
+huggingface-hub==0.33.0
+idna==3.10
+Jinja2==3.1.6
+MarkupSafe==3.0.2
+mpmath==1.3.0
+multidict==6.4.4
+multiprocess==0.70.16
+networkx==3.5
+numpy==2.3.0
+packaging==25.0
+pandas==2.3.0
+pillow==11.2.1
+pip==25.1.1
+platformdirs==4.3.8
+propcache==0.3.2
+protobuf==6.31.1
+psutil==7.0.0
+pyarrow==20.0.0
+pydantic==2.11.7
+pydantic_core==2.33.2
+python-dateutil==2.9.0.post0
+pytz==2025.2
+PyYAML==6.0.2
+regex==2024.11.6
+requests==2.32.4
+safetensors==0.5.3
+sentry-sdk==2.30.0
+setproctitle==1.3.6
+setuptools==80.9.0
+six==1.17.0
+smmap==5.0.2
+sympy==1.14.0
+tevatron==0.0.1
+tokenizers==0.21.1
+torch==2.7.1
+torchaudio==2.7.1
+torchvision==0.22.1
+tqdm==4.67.1
+transformers==4.52.4
+typing_extensions==4.14.0
+typing-inspection==0.4.1
+tzdata==2025.2
+urllib3==2.4.0
+wandb==0.20.1
+xxhash==3.5.0
+yarl==1.20.1
+tevatron==0.0.1

wandb/offline-run-20250615_083045-gw7kaqtk/files/wandb-metadata.json

@@ -0,0 +1,101 @@
+{
+  "os":  "Windows-10-10.0.19045-SP0",
+  "python":  "CPython 3.13.5",
+  "startedAt":  "2025-06-15T01:30:45.974959Z",
+  "args":  [
+    "--output_dir",
+    "logs/test_glen_vault/GLEN_P2_test",
+    "--model_name_or_path",
+    "logs/test_glen_vault/GLEN_P1_test",
+    "--per_device_train_batch_size",
+    "4",
+    "--per_device_eval_batch_size",
+    "2",
+    "--gradient_accumulation_steps",
+    "4",
+    "--dropout_rate",
+    "0.1",
+    "--warmup_ratio",
+    "0.1",
+    "--id_class",
+    "t5_bm25_truncate_3",
+    "--dataset_name",
+    "the_vault",
+    "--test100",
+    "1",
+    "--tree",
+    "1",
+    "--q_max_len",
+    "32",
+    "--p_max_len",
+    "128",
+    "--negative_passage_type",
+    "self",
+    "--positive_passage_no_shuffle",
+    "True",
+    "--tie_word_embeddings",
+    "True",
+    "--num_return_sequences",
+    "5",
+    "--logging_steps",
+    "10",
+    "--overwrite_output_dir",
+    "--wandb_tag",
+    "test_glen_vault_p2",
+    "--do_eval",
+    "False",
+    "--num_train_epochs",
+    "1",
+    "--save_steps",
+    "50",
+    "--save_strategy",
+    "steps",
+    "--evaluation_strategy",
+    "no",
+    "--seed",
+    "42",
+    "--gpu_memory_threshold",
+    "0.85",
+    "--gpu_check_interval",
+    "50",
+    "--fp16",
+    "True"
+  ],
+  "program":  "H:\\Code\\GLEN-model\\examples\\glen_phase2\\train_glen.py",
+  "codePath":  "examples\\glen_phase2\\train_glen.py",
+  "git":  {
+    "remote":  "https://QuanTH02:@huggingface.co/QuanTH02/GLEN-model",
+    "commit":  "6534252bf5ad60b20ba58d7d578a982aabeaacaa"
+  },
+  "root":  "H:\\Code\\GLEN-model",
+  "host":  "FPS-33",
+  "executable":  "H:\\Code\\GLEN-model\\.env\\Scripts\\python.exe",
+  "codePathLocal":  "examples\\glen_phase2\\train_glen.py",
+  "cpu_count":  10,
+  "cpu_count_logical":  16,
+  "gpu":  "NVIDIA GeForce RTX 4060",
+  "gpu_count":  1,
+  "disk":  {
+    "/":  {
+      "total":  "8001561812992",
+      "used":  "3638731177984"
+    }
+  },
+  "memory":  {
+    "total":  "34157170688"
+  },
+  "cpu":  {
+    "count":  10,
+    "countLogical":  16
+  },
+  "gpu_nvidia":  [
+    {
+      "name":  "NVIDIA GeForce RTX 4060",
+      "memoryTotal":  "8585740288",
+      "cudaCores":  3072,
+      "architecture":  "Ada",
+      "uuid":  "GPU-7e0c8403-933a-8533-bde6-f629db871693"
+    }
+  ],
+  "cudaVersion":  "12.8"
+}

wandb/offline-run-20250615_083755-qlx0umrq/files/requirements.txt

@@ -0,0 +1,64 @@
+accelerate==1.7.0
+aiohappyeyeballs==2.6.1
+aiohttp==3.12.13
+aiosignal==1.3.2
+annotated-types==0.7.0
+attrs==25.3.0
+certifi==2025.4.26
+charset-normalizer==3.4.2
+click==8.2.1
+colorama==0.4.6
+datasets==3.6.0
+dill==0.3.8
+filelock==3.18.0
+frozenlist==1.7.0
+fsspec==2025.3.0
+gitdb==4.0.12
+GitPython==3.1.44
+huggingface-hub==0.33.0
+idna==3.10
+Jinja2==3.1.6
+MarkupSafe==3.0.2
+mpmath==1.3.0
+multidict==6.4.4
+multiprocess==0.70.16
+networkx==3.5
+numpy==2.3.0
+packaging==25.0
+pandas==2.3.0
+pillow==11.2.1
+pip==25.1.1
+platformdirs==4.3.8
+propcache==0.3.2
+protobuf==6.31.1
+psutil==7.0.0
+pyarrow==20.0.0
+pydantic==2.11.7
+pydantic_core==2.33.2
+python-dateutil==2.9.0.post0
+pytz==2025.2
+PyYAML==6.0.2
+regex==2024.11.6
+requests==2.32.4
+safetensors==0.5.3
+sentry-sdk==2.30.0
+setproctitle==1.3.6
+setuptools==80.9.0
+six==1.17.0
+smmap==5.0.2
+sympy==1.14.0
+tevatron==0.0.1
+tokenizers==0.21.1
+torch==2.7.1
+torchaudio==2.7.1
+torchvision==0.22.1
+tqdm==4.67.1
+transformers==4.52.4
+typing_extensions==4.14.0
+typing-inspection==0.4.1
+tzdata==2025.2
+urllib3==2.4.0
+wandb==0.20.1
+xxhash==3.5.0
+yarl==1.20.1
+tevatron==0.0.1

wandb/offline-run-20250615_083755-qlx0umrq/files/wandb-metadata.json

@@ -0,0 +1,111 @@
+{
+  "os":  "Windows-10-10.0.19045-SP0",
+  "python":  "CPython 3.13.5",
+  "startedAt":  "2025-06-15T01:37:56.172793Z",
+  "args":  [
+    "--output_dir",
+    "logs/test_glen_vault/GLEN_P1_test",
+    "--model_name_or_path",
+    "t5-base",
+    "--query_type",
+    "gtq_doc",
+    "--per_device_train_batch_size",
+    "8",
+    "--per_device_eval_batch_size",
+    "4",
+    "--gradient_accumulation_steps",
+    "2",
+    "--dropout_rate",
+    "0.1",
+    "--Rdrop",
+    "0.15",
+    "--aug_query",
+    "True",
+    "--aug_query_type",
+    "corrupted_query",
+    "--input_dropout",
+    "1",
+    "--id_class",
+    "t5_bm25_truncate_3",
+    "--dataset_name",
+    "the_vault",
+    "--test100",
+    "1",
+    "--tree",
+    "1",
+    "--pretrain_decoder",
+    "True",
+    "--max_input_length",
+    "128",
+    "--val_check_interval",
+    "1.0",
+    "--tie_word_embeddings",
+    "True",
+    "--decoder_input",
+    "doc_rep",
+    "--max_output_length",
+    "5",
+    "--num_return_sequences",
+    "5",
+    "--logging_steps",
+    "10",
+    "--overwrite_output_dir",
+    "--wandb_tag",
+    "test_glen_vault_p1",
+    "--do_eval",
+    "False",
+    "--num_train_epochs",
+    "1",
+    "--save_steps",
+    "50",
+    "--save_strategy",
+    "steps",
+    "--evaluation_strategy",
+    "no",
+    "--seed",
+    "42",
+    "--gpu_memory_threshold",
+    "0.85",
+    "--gpu_check_interval",
+    "50",
+    "--fp16",
+    "True"
+  ],
+  "program":  "H:\\Code\\GLEN-model\\examples\\glen_phase1\\train_glen.py",
+  "codePath":  "examples\\glen_phase1\\train_glen.py",
+  "git":  {
+    "remote":  "https://QuanTH02:@huggingface.co/QuanTH02/GLEN-model",
+    "commit":  "6534252bf5ad60b20ba58d7d578a982aabeaacaa"
+  },
+  "root":  "H:\\Code\\GLEN-model",
+  "host":  "FPS-33",
+  "executable":  "H:\\Code\\GLEN-model\\.env\\Scripts\\python.exe",
+  "codePathLocal":  "examples\\glen_phase1\\train_glen.py",
+  "cpu_count":  10,
+  "cpu_count_logical":  16,
+  "gpu":  "NVIDIA GeForce RTX 4060",
+  "gpu_count":  1,
+  "disk":  {
+    "/":  {
+      "total":  "8001561812992",
+      "used":  "3639622901760"
+    }
+  },
+  "memory":  {
+    "total":  "34157170688"
+  },
+  "cpu":  {
+    "count":  10,
+    "countLogical":  16
+  },
+  "gpu_nvidia":  [
+    {
+      "name":  "NVIDIA GeForce RTX 4060",
+      "memoryTotal":  "8585740288",
+      "cudaCores":  3072,
+      "architecture":  "Ada",
+      "uuid":  "GPU-7e0c8403-933a-8533-bde6-f629db871693"
+    }
+  ],
+  "cudaVersion":  "12.8"
+}

wandb/offline-run-20250615_084004-v280mta6/files/requirements.txt

@@ -0,0 +1,64 @@
+accelerate==1.7.0
+aiohappyeyeballs==2.6.1
+aiohttp==3.12.13
+aiosignal==1.3.2
+annotated-types==0.7.0
+attrs==25.3.0
+certifi==2025.4.26
+charset-normalizer==3.4.2
+click==8.2.1
+colorama==0.4.6
+datasets==3.6.0
+dill==0.3.8
+filelock==3.18.0
+frozenlist==1.7.0
+fsspec==2025.3.0
+gitdb==4.0.12
+GitPython==3.1.44
+huggingface-hub==0.33.0
+idna==3.10
+Jinja2==3.1.6
+MarkupSafe==3.0.2
+mpmath==1.3.0
+multidict==6.4.4
+multiprocess==0.70.16
+networkx==3.5
+numpy==2.3.0
+packaging==25.0
+pandas==2.3.0
+pillow==11.2.1
+pip==25.1.1
+platformdirs==4.3.8
+propcache==0.3.2
+protobuf==6.31.1
+psutil==7.0.0
+pyarrow==20.0.0
+pydantic==2.11.7
+pydantic_core==2.33.2
+python-dateutil==2.9.0.post0
+pytz==2025.2
+PyYAML==6.0.2
+regex==2024.11.6
+requests==2.32.4
+safetensors==0.5.3
+sentry-sdk==2.30.0
+setproctitle==1.3.6
+setuptools==80.9.0
+six==1.17.0
+smmap==5.0.2
+sympy==1.14.0
+tevatron==0.0.1
+tokenizers==0.21.1
+torch==2.7.1
+torchaudio==2.7.1
+torchvision==0.22.1
+tqdm==4.67.1
+transformers==4.52.4
+typing_extensions==4.14.0
+typing-inspection==0.4.1
+tzdata==2025.2
+urllib3==2.4.0
+wandb==0.20.1
+xxhash==3.5.0
+yarl==1.20.1
+tevatron==0.0.1

wandb/offline-run-20250615_084004-v280mta6/files/wandb-metadata.json

@@ -0,0 +1,101 @@
+{
+  "os":  "Windows-10-10.0.19045-SP0",
+  "python":  "CPython 3.13.5",
+  "startedAt":  "2025-06-15T01:40:04.662871Z",
+  "args":  [
+    "--output_dir",
+    "logs/test_glen_vault/GLEN_P2_test",
+    "--model_name_or_path",
+    "logs/test_glen_vault/GLEN_P1_test",
+    "--per_device_train_batch_size",
+    "4",
+    "--per_device_eval_batch_size",
+    "2",
+    "--gradient_accumulation_steps",
+    "4",
+    "--dropout_rate",
+    "0.1",
+    "--warmup_ratio",
+    "0.1",
+    "--id_class",
+    "t5_bm25_truncate_3",
+    "--dataset_name",
+    "the_vault",
+    "--test100",
+    "1",
+    "--tree",
+    "1",
+    "--q_max_len",
+    "32",
+    "--p_max_len",
+    "128",
+    "--negative_passage_type",
+    "self",
+    "--positive_passage_no_shuffle",
+    "True",
+    "--tie_word_embeddings",
+    "True",
+    "--num_return_sequences",
+    "5",
+    "--logging_steps",
+    "10",
+    "--overwrite_output_dir",
+    "--wandb_tag",
+    "test_glen_vault_p2",
+    "--do_eval",
+    "False",
+    "--num_train_epochs",
+    "1",
+    "--save_steps",
+    "50",
+    "--save_strategy",
+    "steps",
+    "--evaluation_strategy",
+    "no",
+    "--seed",
+    "42",
+    "--gpu_memory_threshold",
+    "0.85",
+    "--gpu_check_interval",
+    "50",
+    "--fp16",
+    "True"
+  ],
+  "program":  "H:\\Code\\GLEN-model\\examples\\glen_phase2\\train_glen.py",
+  "codePath":  "examples\\glen_phase2\\train_glen.py",
+  "git":  {
+    "remote":  "https://QuanTH02:@huggingface.co/QuanTH02/GLEN-model",
+    "commit":  "6534252bf5ad60b20ba58d7d578a982aabeaacaa"
+  },
+  "root":  "H:\\Code\\GLEN-model",
+  "host":  "FPS-33",
+  "executable":  "H:\\Code\\GLEN-model\\.env\\Scripts\\python.exe",
+  "codePathLocal":  "examples\\glen_phase2\\train_glen.py",
+  "cpu_count":  10,
+  "cpu_count_logical":  16,
+  "gpu":  "NVIDIA GeForce RTX 4060",
+  "gpu_count":  1,
+  "disk":  {
+    "/":  {
+      "total":  "8001561812992",
+      "used":  "3640601427968"
+    }
+  },
+  "memory":  {
+    "total":  "34157170688"
+  },
+  "cpu":  {
+    "count":  10,
+    "countLogical":  16
+  },
+  "gpu_nvidia":  [
+    {
+      "name":  "NVIDIA GeForce RTX 4060",
+      "memoryTotal":  "8585740288",
+      "cudaCores":  3072,
+      "architecture":  "Ada",
+      "uuid":  "GPU-7e0c8403-933a-8533-bde6-f629db871693"
+    }
+  ],
+  "cudaVersion":  "12.8"
+}

wandb/offline-run-20250615_084743-xvd6hiwa/files/requirements.txt

@@ -0,0 +1,64 @@
+accelerate==1.7.0
+aiohappyeyeballs==2.6.1
+aiohttp==3.12.13
+aiosignal==1.3.2
+annotated-types==0.7.0
+attrs==25.3.0
+certifi==2025.4.26
+charset-normalizer==3.4.2
+click==8.2.1
+colorama==0.4.6
+datasets==3.6.0
+dill==0.3.8
+filelock==3.18.0
+frozenlist==1.7.0
+fsspec==2025.3.0
+gitdb==4.0.12
+GitPython==3.1.44
+huggingface-hub==0.33.0
+idna==3.10
+Jinja2==3.1.6
+MarkupSafe==3.0.2
+mpmath==1.3.0
+multidict==6.4.4
+multiprocess==0.70.16
+networkx==3.5
+numpy==2.3.0
+packaging==25.0
+pandas==2.3.0
+pillow==11.2.1
+pip==25.1.1
+platformdirs==4.3.8
+propcache==0.3.2
+protobuf==6.31.1
+psutil==7.0.0
+pyarrow==20.0.0
+pydantic==2.11.7
+pydantic_core==2.33.2
+python-dateutil==2.9.0.post0
+pytz==2025.2
+PyYAML==6.0.2
+regex==2024.11.6
+requests==2.32.4
+safetensors==0.5.3
+sentry-sdk==2.30.0
+setproctitle==1.3.6
+setuptools==80.9.0
+six==1.17.0
+smmap==5.0.2
+sympy==1.14.0
+tevatron==0.0.1
+tokenizers==0.21.1
+torch==2.7.1
+torchaudio==2.7.1
+torchvision==0.22.1
+tqdm==4.67.1
+transformers==4.52.4
+typing_extensions==4.14.0
+typing-inspection==0.4.1
+tzdata==2025.2
+urllib3==2.4.0
+wandb==0.20.1
+xxhash==3.5.0
+yarl==1.20.1
+tevatron==0.0.1

wandb/offline-run-20250615_084743-xvd6hiwa/files/wandb-metadata.json

@@ -0,0 +1,111 @@
+{
+  "os":  "Windows-10-10.0.19045-SP0",
+  "python":  "CPython 3.13.5",
+  "startedAt":  "2025-06-15T01:47:43.951676Z",
+  "args":  [
+    "--output_dir",
+    "logs/test_glen_vault/GLEN_P1_test",
+    "--model_name_or_path",
+    "t5-base",
+    "--query_type",
+    "gtq_doc",
+    "--per_device_train_batch_size",
+    "8",
+    "--per_device_eval_batch_size",
+    "4",
+    "--gradient_accumulation_steps",
+    "2",
+    "--dropout_rate",
+    "0.1",
+    "--Rdrop",
+    "0.15",
+    "--aug_query",
+    "True",
+    "--aug_query_type",
+    "corrupted_query",
+    "--input_dropout",
+    "1",
+    "--id_class",
+    "t5_bm25_truncate_3",
+    "--dataset_name",
+    "the_vault",
+    "--test100",
+    "1",
+    "--tree",
+    "1",
+    "--pretrain_decoder",
+    "True",
+    "--max_input_length",
+    "128",
+    "--val_check_interval",
+    "1.0",
+    "--tie_word_embeddings",
+    "True",
+    "--decoder_input",
+    "doc_rep",
+    "--max_output_length",
+    "5",
+    "--num_return_sequences",
+    "5",
+    "--logging_steps",
+    "10",
+    "--overwrite_output_dir",
+    "--wandb_tag",
+    "test_glen_vault_p1",
+    "--do_eval",
+    "False",
+    "--num_train_epochs",
+    "1",
+    "--save_steps",
+    "50",
+    "--save_strategy",
+    "steps",
+    "--evaluation_strategy",
+    "no",
+    "--seed",
+    "42",
+    "--gpu_memory_threshold",
+    "0.85",
+    "--gpu_check_interval",
+    "50",
+    "--fp16",
+    "True"
+  ],
+  "program":  "H:\\Code\\GLEN-model\\examples\\glen_phase1\\train_glen.py",
+  "codePath":  "examples\\glen_phase1\\train_glen.py",
+  "git":  {
+    "remote":  "https://QuanTH02:@huggingface.co/QuanTH02/GLEN-model",
+    "commit":  "6534252bf5ad60b20ba58d7d578a982aabeaacaa"
+  },
+  "root":  "H:\\Code\\GLEN-model",
+  "host":  "FPS-33",
+  "executable":  "H:\\Code\\GLEN-model\\.env\\Scripts\\python.exe",
+  "codePathLocal":  "examples\\glen_phase1\\train_glen.py",
+  "cpu_count":  10,
+  "cpu_count_logical":  16,
+  "gpu":  "NVIDIA GeForce RTX 4060",
+  "gpu_count":  1,
+  "disk":  {
+    "/":  {
+      "total":  "8001561812992",
+      "used":  "3640081137664"
+    }
+  },
+  "memory":  {
+    "total":  "34157170688"
+  },
+  "cpu":  {
+    "count":  10,
+    "countLogical":  16
+  },
+  "gpu_nvidia":  [
+    {
+      "name":  "NVIDIA GeForce RTX 4060",
+      "memoryTotal":  "8585740288",
+      "cudaCores":  3072,
+      "architecture":  "Ada",
+      "uuid":  "GPU-7e0c8403-933a-8533-bde6-f629db871693"
+    }
+  ],
+  "cudaVersion":  "12.8"
+}

wandb/offline-run-20250615_085008-fr23ohzz/files/requirements.txt

@@ -0,0 +1,64 @@
+accelerate==1.7.0
+aiohappyeyeballs==2.6.1
+aiohttp==3.12.13
+aiosignal==1.3.2
+annotated-types==0.7.0
+attrs==25.3.0
+certifi==2025.4.26
+charset-normalizer==3.4.2
+click==8.2.1
+colorama==0.4.6
+datasets==3.6.0
+dill==0.3.8
+filelock==3.18.0
+frozenlist==1.7.0
+fsspec==2025.3.0
+gitdb==4.0.12
+GitPython==3.1.44
+huggingface-hub==0.33.0
+idna==3.10
+Jinja2==3.1.6
+MarkupSafe==3.0.2
+mpmath==1.3.0
+multidict==6.4.4
+multiprocess==0.70.16
+networkx==3.5
+numpy==2.3.0
+packaging==25.0
+pandas==2.3.0
+pillow==11.2.1
+pip==25.1.1
+platformdirs==4.3.8
+propcache==0.3.2
+protobuf==6.31.1
+psutil==7.0.0
+pyarrow==20.0.0
+pydantic==2.11.7
+pydantic_core==2.33.2
+python-dateutil==2.9.0.post0
+pytz==2025.2
+PyYAML==6.0.2
+regex==2024.11.6
+requests==2.32.4
+safetensors==0.5.3
+sentry-sdk==2.30.0
+setproctitle==1.3.6
+setuptools==80.9.0
+six==1.17.0
+smmap==5.0.2
+sympy==1.14.0
+tevatron==0.0.1
+tokenizers==0.21.1
+torch==2.7.1
+torchaudio==2.7.1
+torchvision==0.22.1
+tqdm==4.67.1
+transformers==4.52.4
+typing_extensions==4.14.0
+typing-inspection==0.4.1
+tzdata==2025.2
+urllib3==2.4.0
+wandb==0.20.1
+xxhash==3.5.0
+yarl==1.20.1
+tevatron==0.0.1

wandb/offline-run-20250615_085008-fr23ohzz/files/wandb-metadata.json

@@ -0,0 +1,101 @@
+{
+  "os":  "Windows-10-10.0.19045-SP0",
+  "python":  "CPython 3.13.5",
+  "startedAt":  "2025-06-15T01:50:09.342451Z",
+  "args":  [
+    "--output_dir",
+    "logs/test_glen_vault/GLEN_P2_test",
+    "--model_name_or_path",
+    "logs/test_glen_vault/GLEN_P1_test",
+    "--per_device_train_batch_size",
+    "4",
+    "--per_device_eval_batch_size",
+    "2",
+    "--gradient_accumulation_steps",
+    "4",
+    "--dropout_rate",
+    "0.1",
+    "--warmup_ratio",
+    "0.1",
+    "--id_class",
+    "t5_bm25_truncate_3",
+    "--dataset_name",
+    "the_vault",
+    "--test100",
+    "1",
+    "--tree",
+    "1",
+    "--q_max_len",
+    "32",
+    "--p_max_len",
+    "128",
+    "--negative_passage_type",
+    "self",
+    "--positive_passage_no_shuffle",
+    "True",
+    "--tie_word_embeddings",
+    "True",
+    "--num_return_sequences",
+    "5",
+    "--logging_steps",
+    "10",
+    "--overwrite_output_dir",
+    "--wandb_tag",
+    "test_glen_vault_p2",
+    "--do_eval",
+    "False",
+    "--num_train_epochs",
+    "1",
+    "--save_steps",
+    "50",
+    "--save_strategy",
+    "steps",
+    "--evaluation_strategy",
+    "no",
+    "--seed",
+    "42",
+    "--gpu_memory_threshold",
+    "0.85",
+    "--gpu_check_interval",
+    "50",
+    "--fp16",
+    "True"
+  ],
+  "program":  "H:\\Code\\GLEN-model\\examples\\glen_phase2\\train_glen.py",
+  "codePath":  "examples\\glen_phase2\\train_glen.py",
+  "git":  {
+    "remote":  "https://QuanTH02:@huggingface.co/QuanTH02/GLEN-model",
+    "commit":  "6534252bf5ad60b20ba58d7d578a982aabeaacaa"
+  },
+  "root":  "H:\\Code\\GLEN-model",
+  "host":  "FPS-33",
+  "executable":  "H:\\Code\\GLEN-model\\.env\\Scripts\\python.exe",
+  "codePathLocal":  "examples\\glen_phase2\\train_glen.py",
+  "cpu_count":  10,
+  "cpu_count_logical":  16,
+  "gpu":  "NVIDIA GeForce RTX 4060",
+  "gpu_count":  1,
+  "disk":  {
+    "/":  {
+      "total":  "8001561812992",
+      "used":  "3640533409792"
+    }
+  },
+  "memory":  {
+    "total":  "34157170688"
+  },
+  "cpu":  {
+    "count":  10,
+    "countLogical":  16
+  },
+  "gpu_nvidia":  [
+    {
+      "name":  "NVIDIA GeForce RTX 4060",
+      "memoryTotal":  "8585740288",
+      "cudaCores":  3072,
+      "architecture":  "Ada",
+      "uuid":  "GPU-7e0c8403-933a-8533-bde6-f629db871693"
+    }
+  ],
+  "cudaVersion":  "12.8"
+}

wandb/offline-run-20250615_085636-ufk3qyrh/files/requirements.txt

@@ -0,0 +1,64 @@
+accelerate==1.7.0
+aiohappyeyeballs==2.6.1
+aiohttp==3.12.13
+aiosignal==1.3.2
+annotated-types==0.7.0
+attrs==25.3.0
+certifi==2025.4.26
+charset-normalizer==3.4.2
+click==8.2.1
+colorama==0.4.6
+datasets==3.6.0
+dill==0.3.8
+filelock==3.18.0
+frozenlist==1.7.0
+fsspec==2025.3.0
+gitdb==4.0.12
+GitPython==3.1.44
+huggingface-hub==0.33.0
+idna==3.10
+Jinja2==3.1.6
+MarkupSafe==3.0.2
+mpmath==1.3.0
+multidict==6.4.4
+multiprocess==0.70.16
+networkx==3.5
+numpy==2.3.0
+packaging==25.0
+pandas==2.3.0
+pillow==11.2.1
+pip==25.1.1
+platformdirs==4.3.8
+propcache==0.3.2
+protobuf==6.31.1
+psutil==7.0.0
+pyarrow==20.0.0
+pydantic==2.11.7
+pydantic_core==2.33.2
+python-dateutil==2.9.0.post0
+pytz==2025.2
+PyYAML==6.0.2
+regex==2024.11.6
+requests==2.32.4
+safetensors==0.5.3
+sentry-sdk==2.30.0
+setproctitle==1.3.6
+setuptools==80.9.0
+six==1.17.0
+smmap==5.0.2
+sympy==1.14.0
+tevatron==0.0.1
+tokenizers==0.21.1
+torch==2.7.1
+torchaudio==2.7.1
+torchvision==0.22.1
+tqdm==4.67.1
+transformers==4.52.4
+typing_extensions==4.14.0
+typing-inspection==0.4.1
+tzdata==2025.2
+urllib3==2.4.0
+wandb==0.20.1
+xxhash==3.5.0
+yarl==1.20.1
+tevatron==0.0.1

wandb/offline-run-20250615_085636-ufk3qyrh/files/wandb-metadata.json

@@ -0,0 +1,113 @@
+{
+  "os":  "Windows-10-10.0.19045-SP0",
+  "python":  "CPython 3.13.5",
+  "startedAt":  "2025-06-15T01:56:36.587828Z",
+  "args":  [
+    "--output_dir",
+    "logs/test_glen_vault/GLEN_P1_test",
+    "--model_name_or_path",
+    "t5-base",
+    "--query_type",
+    "gtq_doc",
+    "--per_device_train_batch_size",
+    "8",
+    "--per_device_eval_batch_size",
+    "4",
+    "--gradient_accumulation_steps",
+    "2",
+    "--dropout_rate",
+    "0.1",
+    "--Rdrop",
+    "0.15",
+    "--aug_query",
+    "True",
+    "--aug_query_type",
+    "corrupted_query",
+    "--input_dropout",
+    "1",
+    "--id_class",
+    "t5_bm25_truncate_3",
+    "--dataset_name",
+    "the_vault",
+    "--test100",
+    "1",
+    "--tree",
+    "1",
+    "--pretrain_decoder",
+    "True",
+    "--max_input_length",
+    "128",
+    "--val_check_interval",
+    "1.0",
+    "--tie_word_embeddings",
+    "True",
+    "--decoder_input",
+    "doc_rep",
+    "--max_output_length",
+    "5",
+    "--num_return_sequences",
+    "5",
+    "--logging_steps",
+    "100",
+    "--overwrite_output_dir",
+    "--wandb_tag",
+    "glen_vault_test_p1",
+    "--do_eval",
+    "True",
+    "--num_train_epochs",
+    "1",
+    "--save_steps",
+    "1000",
+    "--save_strategy",
+    "steps",
+    "--evaluation_strategy",
+    "steps",
+    "--eval_steps",
+    "1000",
+    "--seed",
+    "42",
+    "--gpu_memory_threshold",
+    "0.85",
+    "--gpu_check_interval",
+    "50",
+    "--fp16",
+    "True"
+  ],
+  "program":  "H:\\Code\\GLEN-model\\examples\\glen_phase1\\train_glen.py",
+  "codePath":  "examples\\glen_phase1\\train_glen.py",
+  "git":  {
+    "remote":  "https://QuanTH02:@huggingface.co/QuanTH02/GLEN-model",
+    "commit":  "6534252bf5ad60b20ba58d7d578a982aabeaacaa"
+  },
+  "root":  "H:\\Code\\GLEN-model",
+  "host":  "FPS-33",
+  "executable":  "H:\\Code\\GLEN-model\\.env\\Scripts\\python.exe",
+  "codePathLocal":  "examples\\glen_phase1\\train_glen.py",
+  "cpu_count":  10,
+  "cpu_count_logical":  16,
+  "gpu":  "NVIDIA GeForce RTX 4060",
+  "gpu_count":  1,
+  "disk":  {
+    "/":  {
+      "total":  "8001561812992",
+      "used":  "3640026095616"
+    }
+  },
+  "memory":  {
+    "total":  "34157170688"
+  },
+  "cpu":  {
+    "count":  10,
+    "countLogical":  16
+  },
+  "gpu_nvidia":  [
+    {
+      "name":  "NVIDIA GeForce RTX 4060",
+      "memoryTotal":  "8585740288",
+      "cudaCores":  3072,
+      "architecture":  "Ada",
+      "uuid":  "GPU-7e0c8403-933a-8533-bde6-f629db871693"
+    }
+  ],
+  "cudaVersion":  "12.8"
+}

wandb/offline-run-20250615_090510-p2obgs7h/files/requirements.txt

@@ -0,0 +1,64 @@
+accelerate==1.7.0
+aiohappyeyeballs==2.6.1
+aiohttp==3.12.13
+aiosignal==1.3.2
+annotated-types==0.7.0
+attrs==25.3.0
+certifi==2025.4.26
+charset-normalizer==3.4.2
+click==8.2.1
+colorama==0.4.6
+datasets==3.6.0
+dill==0.3.8
+filelock==3.18.0
+frozenlist==1.7.0
+fsspec==2025.3.0
+gitdb==4.0.12
+GitPython==3.1.44
+huggingface-hub==0.33.0
+idna==3.10
+Jinja2==3.1.6
+MarkupSafe==3.0.2
+mpmath==1.3.0
+multidict==6.4.4
+multiprocess==0.70.16
+networkx==3.5
+numpy==2.3.0
+packaging==25.0
+pandas==2.3.0
+pillow==11.2.1
+pip==25.1.1
+platformdirs==4.3.8
+propcache==0.3.2
+protobuf==6.31.1
+psutil==7.0.0
+pyarrow==20.0.0
+pydantic==2.11.7
+pydantic_core==2.33.2
+python-dateutil==2.9.0.post0
+pytz==2025.2
+PyYAML==6.0.2
+regex==2024.11.6
+requests==2.32.4
+safetensors==0.5.3
+sentry-sdk==2.30.0
+setproctitle==1.3.6
+setuptools==80.9.0
+six==1.17.0
+smmap==5.0.2
+sympy==1.14.0
+tevatron==0.0.1
+tokenizers==0.21.1
+torch==2.7.1
+torchaudio==2.7.1
+torchvision==0.22.1
+tqdm==4.67.1
+transformers==4.52.4
+typing_extensions==4.14.0
+typing-inspection==0.4.1
+tzdata==2025.2
+urllib3==2.4.0
+wandb==0.20.1
+xxhash==3.5.0
+yarl==1.20.1
+tevatron==0.0.1

wandb/offline-run-20250615_090510-p2obgs7h/files/wandb-metadata.json

@@ -0,0 +1,113 @@
+{
+  "os":  "Windows-10-10.0.19045-SP0",
+  "python":  "CPython 3.13.5",
+  "startedAt":  "2025-06-15T02:05:11.108383Z",
+  "args":  [
+    "--output_dir",
+    "logs/test_glen_vault/GLEN_P1_test",
+    "--model_name_or_path",
+    "t5-base",
+    "--query_type",
+    "gtq_doc",
+    "--per_device_train_batch_size",
+    "8",
+    "--per_device_eval_batch_size",
+    "4",
+    "--gradient_accumulation_steps",
+    "2",
+    "--dropout_rate",
+    "0.1",
+    "--Rdrop",
+    "0.15",
+    "--aug_query",
+    "True",
+    "--aug_query_type",
+    "corrupted_query",
+    "--input_dropout",
+    "1",
+    "--id_class",
+    "t5_bm25_truncate_3",
+    "--dataset_name",
+    "the_vault",
+    "--test100",
+    "1",
+    "--tree",
+    "1",
+    "--pretrain_decoder",
+    "True",
+    "--max_input_length",
+    "128",
+    "--val_check_interval",
+    "1.0",
+    "--tie_word_embeddings",
+    "True",
+    "--decoder_input",
+    "doc_rep",
+    "--max_output_length",
+    "5",
+    "--num_return_sequences",
+    "5",
+    "--logging_steps",
+    "100",
+    "--overwrite_output_dir",
+    "--wandb_tag",
+    "glen_vault_test_p1",
+    "--do_eval",
+    "True",
+    "--num_train_epochs",
+    "1",
+    "--save_steps",
+    "1000",
+    "--save_strategy",
+    "steps",
+    "--evaluation_strategy",
+    "steps",
+    "--eval_steps",
+    "1000",
+    "--seed",
+    "42",
+    "--gpu_memory_threshold",
+    "0.85",
+    "--gpu_check_interval",
+    "50",
+    "--fp16",
+    "True"
+  ],
+  "program":  "H:\\Code\\GLEN-model\\examples\\glen_phase1\\train_glen.py",
+  "codePath":  "examples\\glen_phase1\\train_glen.py",
+  "git":  {
+    "remote":  "https://QuanTH02:@huggingface.co/QuanTH02/GLEN-model",
+    "commit":  "ca9706f426fc8d43aa09c19ad7ec61380c5f7749"
+  },
+  "root":  "H:\\Code\\GLEN-model",
+  "host":  "FPS-33",
+  "executable":  "H:\\Code\\GLEN-model\\.env\\Scripts\\python.exe",
+  "codePathLocal":  "examples\\glen_phase1\\train_glen.py",
+  "cpu_count":  10,
+  "cpu_count_logical":  16,
+  "gpu":  "NVIDIA GeForce RTX 4060",
+  "gpu_count":  1,
+  "disk":  {
+    "/":  {
+      "total":  "8001561812992",
+      "used":  "3639623524352"
+    }
+  },
+  "memory":  {
+    "total":  "34157170688"
+  },
+  "cpu":  {
+    "count":  10,
+    "countLogical":  16
+  },
+  "gpu_nvidia":  [
+    {
+      "name":  "NVIDIA GeForce RTX 4060",
+      "memoryTotal":  "8585740288",
+      "cudaCores":  3072,
+      "architecture":  "Ada",
+      "uuid":  "GPU-7e0c8403-933a-8533-bde6-f629db871693"
+    }
+  ],
+  "cudaVersion":  "12.8"
+}

wandb/offline-run-20250615_090639-ovkkgdmi/files/requirements.txt

@@ -0,0 +1,64 @@
+accelerate==1.7.0
+aiohappyeyeballs==2.6.1
+aiohttp==3.12.13
+aiosignal==1.3.2
+annotated-types==0.7.0
+attrs==25.3.0
+certifi==2025.4.26
+charset-normalizer==3.4.2
+click==8.2.1
+colorama==0.4.6
+datasets==3.6.0
+dill==0.3.8
+filelock==3.18.0
+frozenlist==1.7.0
+fsspec==2025.3.0
+gitdb==4.0.12
+GitPython==3.1.44
+huggingface-hub==0.33.0
+idna==3.10
+Jinja2==3.1.6
+MarkupSafe==3.0.2
+mpmath==1.3.0
+multidict==6.4.4
+multiprocess==0.70.16
+networkx==3.5
+numpy==2.3.0
+packaging==25.0
+pandas==2.3.0
+pillow==11.2.1
+pip==25.1.1
+platformdirs==4.3.8
+propcache==0.3.2
+protobuf==6.31.1
+psutil==7.0.0
+pyarrow==20.0.0
+pydantic==2.11.7
+pydantic_core==2.33.2
+python-dateutil==2.9.0.post0
+pytz==2025.2
+PyYAML==6.0.2
+regex==2024.11.6
+requests==2.32.4
+safetensors==0.5.3
+sentry-sdk==2.30.0
+setproctitle==1.3.6
+setuptools==80.9.0
+six==1.17.0
+smmap==5.0.2
+sympy==1.14.0
+tevatron==0.0.1
+tokenizers==0.21.1
+torch==2.7.1
+torchaudio==2.7.1
+torchvision==0.22.1
+tqdm==4.67.1
+transformers==4.52.4
+typing_extensions==4.14.0
+typing-inspection==0.4.1
+tzdata==2025.2
+urllib3==2.4.0
+wandb==0.20.1
+xxhash==3.5.0
+yarl==1.20.1
+tevatron==0.0.1

wandb/offline-run-20250615_090639-ovkkgdmi/files/wandb-metadata.json

@@ -0,0 +1,101 @@
+{
+  "os":  "Windows-10-10.0.19045-SP0",
+  "python":  "CPython 3.13.5",
+  "startedAt":  "2025-06-15T02:06:40.118965Z",
+  "args":  [
+    "--output_dir",
+    "logs/test_glen_vault/GLEN_P2_test",
+    "--model_name_or_path",
+    "logs/test_glen_vault/GLEN_P1_test",
+    "--per_device_train_batch_size",
+    "4",
+    "--per_device_eval_batch_size",
+    "2",
+    "--gradient_accumulation_steps",
+    "4",
+    "--dropout_rate",
+    "0.1",
+    "--warmup_ratio",
+    "0.1",
+    "--id_class",
+    "t5_bm25_truncate_3",
+    "--dataset_name",
+    "the_vault",
+    "--tree",
+    "1",
+    "--q_max_len",
+    "32",
+    "--p_max_len",
+    "128",
+    "--negative_passage_type",
+    "self",
+    "--positive_passage_no_shuffle",
+    "True",
+    "--tie_word_embeddings",
+    "True",
+    "--num_return_sequences",
+    "5",
+    "--logging_steps",
+    "100",
+    "--overwrite_output_dir",
+    "--wandb_tag",
+    "glen_vault_test_p2",
+    "--do_eval",
+    "True",
+    "--num_train_epochs",
+    "1",
+    "--save_steps",
+    "1000",
+    "--save_strategy",
+    "steps",
+    "--evaluation_strategy",
+    "steps",
+    "--eval_steps",
+    "1000",
+    "--seed",
+    "42",
+    "--gpu_memory_threshold",
+    "0.85",
+    "--gpu_check_interval",
+    "50",
+    "--fp16",
+    "True"
+  ],
+  "program":  "H:\\Code\\GLEN-model\\examples\\glen_phase2\\train_glen.py",
+  "codePath":  "examples\\glen_phase2\\train_glen.py",
+  "git":  {
+    "remote":  "https://QuanTH02:@huggingface.co/QuanTH02/GLEN-model",
+    "commit":  "ca9706f426fc8d43aa09c19ad7ec61380c5f7749"
+  },
+  "root":  "H:\\Code\\GLEN-model",
+  "host":  "FPS-33",
+  "executable":  "H:\\Code\\GLEN-model\\.env\\Scripts\\python.exe",
+  "codePathLocal":  "examples\\glen_phase2\\train_glen.py",
+  "cpu_count":  10,
+  "cpu_count_logical":  16,
+  "gpu":  "NVIDIA GeForce RTX 4060",
+  "gpu_count":  1,
+  "disk":  {
+    "/":  {
+      "total":  "8001561812992",
+      "used":  "3639623598080"
+    }
+  },
+  "memory":  {
+    "total":  "34157170688"
+  },
+  "cpu":  {
+    "count":  10,
+    "countLogical":  16
+  },
+  "gpu_nvidia":  [
+    {
+      "name":  "NVIDIA GeForce RTX 4060",
+      "memoryTotal":  "8585740288",
+      "cudaCores":  3072,
+      "architecture":  "Ada",
+      "uuid":  "GPU-7e0c8403-933a-8533-bde6-f629db871693"
+    }
+  ],
+  "cudaVersion":  "12.8"
+}

wandb/offline-run-20250615_092539-8n51qf7g/files/requirements.txt

@@ -0,0 +1,64 @@
+accelerate==1.7.0
+aiohappyeyeballs==2.6.1
+aiohttp==3.12.13
+aiosignal==1.3.2
+annotated-types==0.7.0
+attrs==25.3.0
+certifi==2025.4.26
+charset-normalizer==3.4.2
+click==8.2.1
+colorama==0.4.6
+datasets==3.6.0
+dill==0.3.8
+filelock==3.18.0
+frozenlist==1.7.0
+fsspec==2025.3.0
+gitdb==4.0.12
+GitPython==3.1.44
+huggingface-hub==0.33.0
+idna==3.10
+Jinja2==3.1.6
+MarkupSafe==3.0.2
+mpmath==1.3.0
+multidict==6.4.4
+multiprocess==0.70.16
+networkx==3.5
+numpy==2.3.0
+packaging==25.0
+pandas==2.3.0
+pillow==11.2.1
+pip==25.1.1
+platformdirs==4.3.8
+propcache==0.3.2
+protobuf==6.31.1
+psutil==7.0.0
+pyarrow==20.0.0
+pydantic==2.11.7
+pydantic_core==2.33.2
+python-dateutil==2.9.0.post0
+pytz==2025.2
+PyYAML==6.0.2
+regex==2024.11.6
+requests==2.32.4
+safetensors==0.5.3
+sentry-sdk==2.30.0
+setproctitle==1.3.6
+setuptools==80.9.0
+six==1.17.0
+smmap==5.0.2
+sympy==1.14.0
+tevatron==0.0.1
+tokenizers==0.21.1
+torch==2.7.1
+torchaudio==2.7.1
+torchvision==0.22.1
+tqdm==4.67.1
+transformers==4.52.4
+typing_extensions==4.14.0
+typing-inspection==0.4.1
+tzdata==2025.2
+urllib3==2.4.0
+wandb==0.20.1
+xxhash==3.5.0
+yarl==1.20.1
+tevatron==0.0.1

wandb/offline-run-20250615_092539-8n51qf7g/files/wandb-metadata.json

@@ -0,0 +1,113 @@
+{
+  "os":  "Windows-10-10.0.19045-SP0",
+  "python":  "CPython 3.13.5",
+  "startedAt":  "2025-06-15T02:25:39.486198Z",
+  "args":  [
+    "--output_dir",
+    "logs/test_glen_vault/GLEN_P1_test",
+    "--model_name_or_path",
+    "t5-base",
+    "--query_type",
+    "gtq_doc",
+    "--per_device_train_batch_size",
+    "8",
+    "--per_device_eval_batch_size",
+    "4",
+    "--gradient_accumulation_steps",
+    "2",
+    "--dropout_rate",
+    "0.1",
+    "--Rdrop",
+    "0.15",
+    "--aug_query",
+    "True",
+    "--aug_query_type",
+    "corrupted_query",
+    "--input_dropout",
+    "1",
+    "--id_class",
+    "t5_bm25_truncate_3",
+    "--dataset_name",
+    "the_vault",
+    "--test100",
+    "1",
+    "--tree",
+    "1",
+    "--pretrain_decoder",
+    "True",
+    "--max_input_length",
+    "128",
+    "--val_check_interval",
+    "1.0",
+    "--tie_word_embeddings",
+    "True",
+    "--decoder_input",
+    "doc_rep",
+    "--max_output_length",
+    "5",
+    "--num_return_sequences",
+    "5",
+    "--logging_steps",
+    "100",
+    "--overwrite_output_dir",
+    "--wandb_tag",
+    "glen_vault_test_p1",
+    "--do_eval",
+    "True",
+    "--num_train_epochs",
+    "1",
+    "--save_steps",
+    "1000",
+    "--save_strategy",
+    "steps",
+    "--evaluation_strategy",
+    "steps",
+    "--eval_steps",
+    "1000",
+    "--seed",
+    "42",
+    "--gpu_memory_threshold",
+    "0.85",
+    "--gpu_check_interval",
+    "50",
+    "--fp16",
+    "True"
+  ],
+  "program":  "H:\\Code\\GLEN-model\\examples\\glen_phase1\\train_glen.py",
+  "codePath":  "examples\\glen_phase1\\train_glen.py",
+  "git":  {
+    "remote":  "https://QuanTH02:@huggingface.co/QuanTH02/GLEN-model",
+    "commit":  "ca9706f426fc8d43aa09c19ad7ec61380c5f7749"
+  },
+  "root":  "H:\\Code\\GLEN-model",
+  "host":  "FPS-33",
+  "executable":  "H:\\Code\\GLEN-model\\.env\\Scripts\\python.exe",
+  "codePathLocal":  "examples\\glen_phase1\\train_glen.py",
+  "cpu_count":  10,
+  "cpu_count_logical":  16,
+  "gpu":  "NVIDIA GeForce RTX 4060",
+  "gpu_count":  1,
+  "disk":  {
+    "/":  {
+      "total":  "8001561812992",
+      "used":  "3639623917568"
+    }
+  },
+  "memory":  {
+    "total":  "34157170688"
+  },
+  "cpu":  {
+    "count":  10,
+    "countLogical":  16
+  },
+  "gpu_nvidia":  [
+    {
+      "name":  "NVIDIA GeForce RTX 4060",
+      "memoryTotal":  "8585740288",
+      "cudaCores":  3072,
+      "architecture":  "Ada",
+      "uuid":  "GPU-7e0c8403-933a-8533-bde6-f629db871693"
+    }
+  ],
+  "cudaVersion":  "12.8"
+}

wandb/offline-run-20250615_092759-cpafuazn/files/requirements.txt

@@ -0,0 +1,64 @@
+accelerate==1.7.0
+aiohappyeyeballs==2.6.1
+aiohttp==3.12.13
+aiosignal==1.3.2
+annotated-types==0.7.0
+attrs==25.3.0
+certifi==2025.4.26
+charset-normalizer==3.4.2
+click==8.2.1
+colorama==0.4.6
+datasets==3.6.0
+dill==0.3.8
+filelock==3.18.0
+frozenlist==1.7.0
+fsspec==2025.3.0
+gitdb==4.0.12
+GitPython==3.1.44
+huggingface-hub==0.33.0
+idna==3.10
+Jinja2==3.1.6
+MarkupSafe==3.0.2
+mpmath==1.3.0
+multidict==6.4.4
+multiprocess==0.70.16
+networkx==3.5
+numpy==2.3.0
+packaging==25.0
+pandas==2.3.0
+pillow==11.2.1
+pip==25.1.1
+platformdirs==4.3.8
+propcache==0.3.2
+protobuf==6.31.1
+psutil==7.0.0
+pyarrow==20.0.0
+pydantic==2.11.7
+pydantic_core==2.33.2
+python-dateutil==2.9.0.post0
+pytz==2025.2
+PyYAML==6.0.2
+regex==2024.11.6
+requests==2.32.4
+safetensors==0.5.3
+sentry-sdk==2.30.0
+setproctitle==1.3.6
+setuptools==80.9.0
+six==1.17.0
+smmap==5.0.2
+sympy==1.14.0
+tevatron==0.0.1
+tokenizers==0.21.1
+torch==2.7.1
+torchaudio==2.7.1
+torchvision==0.22.1
+tqdm==4.67.1
+transformers==4.52.4
+typing_extensions==4.14.0
+typing-inspection==0.4.1
+tzdata==2025.2
+urllib3==2.4.0
+wandb==0.20.1
+xxhash==3.5.0
+yarl==1.20.1
+tevatron==0.0.1

wandb/offline-run-20250615_092759-cpafuazn/files/wandb-metadata.json

@@ -0,0 +1,101 @@
+{
+  "os":  "Windows-10-10.0.19045-SP0",
+  "python":  "CPython 3.13.5",
+  "startedAt":  "2025-06-15T02:28:00.208908Z",
+  "args":  [
+    "--output_dir",
+    "logs/test_glen_vault/GLEN_P2_test",
+    "--model_name_or_path",
+    "logs/test_glen_vault/GLEN_P1_test",
+    "--per_device_train_batch_size",
+    "4",
+    "--per_device_eval_batch_size",
+    "2",
+    "--gradient_accumulation_steps",
+    "4",
+    "--dropout_rate",
+    "0.1",
+    "--warmup_ratio",
+    "0.1",
+    "--id_class",
+    "t5_bm25_truncate_3",
+    "--dataset_name",
+    "the_vault",
+    "--tree",
+    "1",
+    "--q_max_len",
+    "32",
+    "--p_max_len",
+    "128",
+    "--negative_passage_type",
+    "self",
+    "--positive_passage_no_shuffle",
+    "True",
+    "--tie_word_embeddings",
+    "True",
+    "--num_return_sequences",
+    "5",
+    "--logging_steps",
+    "100",
+    "--overwrite_output_dir",
+    "--wandb_tag",
+    "glen_vault_test_p2",
+    "--do_eval",
+    "True",
+    "--num_train_epochs",
+    "1",
+    "--save_steps",
+    "1000",
+    "--save_strategy",
+    "steps",
+    "--evaluation_strategy",
+    "steps",
+    "--eval_steps",
+    "1000",
+    "--seed",
+    "42",
+    "--gpu_memory_threshold",
+    "0.85",
+    "--gpu_check_interval",
+    "50",
+    "--fp16",
+    "True"
+  ],
+  "program":  "H:\\Code\\GLEN-model\\examples\\glen_phase2\\train_glen.py",
+  "codePath":  "examples\\glen_phase2\\train_glen.py",
+  "git":  {
+    "remote":  "https://QuanTH02:@huggingface.co/QuanTH02/GLEN-model",
+    "commit":  "ca9706f426fc8d43aa09c19ad7ec61380c5f7749"
+  },
+  "root":  "H:\\Code\\GLEN-model",
+  "host":  "FPS-33",
+  "executable":  "H:\\Code\\GLEN-model\\.env\\Scripts\\python.exe",
+  "codePathLocal":  "examples\\glen_phase2\\train_glen.py",
+  "cpu_count":  10,
+  "cpu_count_logical":  16,
+  "gpu":  "NVIDIA GeForce RTX 4060",
+  "gpu_count":  1,
+  "disk":  {
+    "/":  {
+      "total":  "8001561812992",
+      "used":  "3639623999488"
+    }
+  },
+  "memory":  {
+    "total":  "34157170688"
+  },
+  "cpu":  {
+    "count":  10,
+    "countLogical":  16
+  },
+  "gpu_nvidia":  [
+    {
+      "name":  "NVIDIA GeForce RTX 4060",
+      "memoryTotal":  "8585740288",
+      "cudaCores":  3072,
+      "architecture":  "Ada",
+      "uuid":  "GPU-7e0c8403-933a-8533-bde6-f629db871693"
+    }
+  ],
+  "cudaVersion":  "12.8"
+}