Mount object storage bucket in sandbox

Each sandbox instance is provisioned with 20GB of ephemeral system disk storage by default for temporary data operations. Upon sandbox termination or timeout, all data within this storage space is automatically purged. Consequently, persistent data requiring long-term retention should be stored in external cloud storage services.

Sandbox storage allocation specifications are subject to modification. Consult the Pricing documentation for current resource allocations and associated costs.

Object Storage represents a highly scalable, durable, and cost-efficient cloud storage architecture offered by major cloud service providers. Within sandbox environments, object storage can be accessed through two primary methodologies: direct programmatic interaction via cloud provider SDKs or CLI utilities, or through FUSE (Filesystem in Userspace) implementations that present object storage buckets as standard POSIX-compliant filesystem mounts.

FUSE (Filesystem in Userspace) is a kernel module and userspace library that enables the implementation of fully functional filesystems in userspace applications. This framework provides an abstraction layer that presents remote cloud storage services as standard filesystem hierarchies, enabling transparent file operations through conventional POSIX interfaces.

This documentation provides comprehensive guidance for integrating object storage buckets from leading cloud service providers into sandbox environments through filesystem mounting techniques.

FUSE-based object storage mounts introduce significant I/O performance overhead due to network latency and protocol translation layers. Applications with stringent performance requirements should avoid this approach. Furthermore, FUSE filesystem operations lack atomicity guarantees inherent to native object storage APIs, creating potential race conditions where local filesystem operations may succeed while corresponding remote operations fail, resulting in data inconsistency.This mounting approach is optimal for read-heavy workloads with infrequent write operations and relaxed performance constraints. For performance-critical applications or frequent write patterns, direct integration using cloud provider SDKs or native REST APIs is strongly recommended.

Amazon S3

Amazon S3 buckets can be mounted as POSIX-compliant filesystems using s3fs-fuse, a FUSE-based filesystem implementation that provides S3 bucket access through standard file operations. The s3fs-fuse package can be integrated during sandbox template creation by incorporating installation commands in the novita.Dockerfile, or installed dynamically within active sandbox instances for ad-hoc requirements. The following novita.Dockerfile demonstrates the integration of s3fs-fuse during template building:

# Compatible with Debian-based distributions
FROM ubuntu:latest

# Critical: s3fs versions below 1.93 contain known mounting issues. Ensure version compatibility.
RUN DEBIAN_FRONTEND=noninteractive apt-get update && apt-get install -y s3fs

The following implementation demonstrates programmatic S3 bucket mounting within sandbox environments using s3fs-fuse:

import { Sandbox } from 'novita-sandbox'

const TEMPLATE_ID = process.env.NOVITA_TEMPLATE_ID
const AWS_ACCESS_KEY_ID = process.env.AWS_ACCESS_KEY_ID
const AWS_SECRET_ACCESS_KEY = process.env.AWS_SECRET_ACCESS_KEY
const AWS_BUCKET_NAME = process.env.AWS_BUCKET_NAME
const AWS_REGION = process.env.AWS_REGION
if (!TEMPLATE_ID || !AWS_ACCESS_KEY_ID || !AWS_SECRET_ACCESS_KEY || !AWS_BUCKET_NAME || !AWS_REGION) {
    throw new Error('Required environment variables not configured: NOVITA_TEMPLATE_ID, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_BUCKET_NAME, AWS_REGION')
}

const MOUNT_DIRECTORY = "/mnt/s3-bucket"

const sandbox = await Sandbox.create(TEMPLATE_ID)

// Initialize mount point directory structure
await sandbox.files.makeDir(MOUNT_DIRECTORY)

// Configure s3fs credentials using standard credential file location
// s3fs-fuse reads AWS credentials from /root/.passwd-s3fs by default
await sandbox.files.write('/root/.passwd-s3fs', `${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}`)

// Enforce secure credential file permissions (owner read-only)
await sandbox.commands.run('sudo chmod 600 /root/.passwd-s3fs')

// Execute S3 bucket mount operation with optimized parameters
// Configuration parameters:
// - allow_other: Enable cross-user filesystem access
// - endpoint: Specify the AWS region endpoint for your bucket
// Reference: https://manpages.ubuntu.com/manpages/noble/en/man1/s3fs.1.html
const mountOptions = `allow_other,endpoint=${AWS_REGION}`
await sandbox.commands.run(`sudo s3fs ${AWS_BUCKET_NAME} ${MOUNT_DIRECTORY} -o ${mountOptions}`)

// Validate mount functionality with write operation
await sandbox.files.write(`${MOUNT_DIRECTORY}/test-file.txt`, 'test-file-content')

// Verify mount integrity through read operation
const content = await sandbox.files.read(`${MOUNT_DIRECTORY}/test-file.txt`)
console.log(content)

await sandbox.kill()

import os
from novita_sandbox.core import Sandbox

TEMPLATE_ID = os.environ.get("NOVITA_TEMPLATE_ID")
AWS_ACCESS_KEY_ID = os.environ.get("AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY = os.environ.get("AWS_SECRET_ACCESS_KEY")
AWS_BUCKET_NAME = os.environ.get("AWS_BUCKET_NAME")
AWS_REGION = os.environ.get("AWS_REGION")
if not TEMPLATE_ID or not AWS_ACCESS_KEY_ID or not AWS_SECRET_ACCESS_KEY or not AWS_BUCKET_NAME or not AWS_REGION:
    raise ValueError("Required environment variables not configured: NOVITA_TEMPLATE_ID, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_BUCKET_NAME, AWS_REGION")

MOUNT_DIRECTORY = "/mnt/s3-bucket"

sandbox = Sandbox.create(TEMPLATE_ID)

# Initialize mount point directory structure
sandbox.files.make_dir(MOUNT_DIRECTORY)

# Configure s3fs credentials using standard credential file location
# s3fs-fuse reads AWS credentials from /root/.passwd-s3fs by default
# Custom credential paths require explicit specification via -o passwd_file parameter
sandbox.files.write("/root/.passwd-s3fs", f"{AWS_ACCESS_KEY_ID}:{AWS_SECRET_ACCESS_KEY}")

# Enforce secure credential file permissions (owner read-only)
sandbox.commands.run("sudo chmod 600 /root/.passwd-s3fs")

# Execute S3 bucket mount operation with optimized parameters
# Configuration parameters:
# - allow_other: Enable cross-user filesystem access
# - endpoint: Specify the AWS region endpoint for your bucket
# Reference: https://manpages.ubuntu.com/manpages/noble/en/man1/s3fs.1.html
mount_options = f"allow_other,endpoint={AWS_REGION}"
sandbox.commands.run(f"sudo s3fs {AWS_BUCKET_NAME} {MOUNT_DIRECTORY} -o {mount_options}")

# Validate mount functionality with write operation
write_result = sandbox.files.write(f"{MOUNT_DIRECTORY}/test-file.txt", "test-file-content")

# Verify mount integrity through read operation
content = sandbox.files.read(f"{MOUNT_DIRECTORY}/test-file.txt")
print("File content:", content)

sandbox.kill()

​Amazon S3

Amazon S3