exploring wnfs

Juan ca6e083277 fixed bug 1 month ago
.cargo b0dc2e01a1 chore: Strip flags 2 (#171) 1 month ago
.github a784caa283 chore(ci)(deps): bump codecov/codecov-action from 3 to 4 (#401) 1 month ago
assets 9145787a42 Documentation of Private Filesystem and Other Cleanups (#68) 1 month ago
proptest-regressions 44b3ae6ef7 more typical dir structure around workspace crates (#73) 1 month ago
scripts ca6e083277 fixed bug 1 month ago
wnfs 6c5f342885 fix: Public reconciliation of a concurrent write and remove (of independent paths) (#432) 1 month ago
wnfs-bench 15fb4c6846 chore: Release version 0.2.0 (#403) 1 month ago
wnfs-common 6c5f342885 fix: Public reconciliation of a concurrent write and remove (of independent paths) (#432) 1 month ago
wnfs-hamt d24f960c53 feat: implement `PublicDirectory` conflict reconciliation (#426) 1 month ago
wnfs-nameaccumulator d24f960c53 feat: implement `PublicDirectory` conflict reconciliation (#426) 1 month ago
wnfs-unixfs-file d24f960c53 feat: implement `PublicDirectory` conflict reconciliation (#426) 1 month ago
wnfs-wasm d58b05fa30 chore(npm)(deps-dev): bump uint8arrays from 5.0.3 to 5.1.0 in /wnfs-wasm (#434) 1 month ago
.gitignore 15b625c172 feat: Implement conflict reconciliation for private directories (#431) 1 month ago
.pre-commit-config.yaml dcbab013d7 chore: put release-please process in place (#96) 1 month ago
.rustfmt.toml 082e3c8b96 refactor: separate into crates (#184) 1 month ago
Cargo.lock 15b625c172 feat: Implement conflict reconciliation for private directories (#431) 1 month ago
Cargo.toml 93eedd4363 feat: Implement `wnfs-unixfs-file` crate for encoding big byte arrays in IPLD (#375) 1 month ago
LICENSE 34e64c1b16 Initial commit 1 month ago
README.md e355c6b60a chore: Actually fix README dep graph 1 month ago
clippy.toml ae476aec1f feat: Switch from Namefilter to Name Accumulators (#247) 1 month ago
codecov.yaml 44b3ae6ef7 more typical dir structure around workspace crates (#73) 1 month ago
deny.toml 553ef2ce38 chore: Use cargo-deny & fix time CVE (#208) 1 month ago

README.md

WNFS Logo

WebNative FileSystem (WNFS)

Concurrency Docs Code Coverage Build Status License Concurrency Docs Discord

:warning: Work in progress :warning:

##

This is a Rust implementation of the WebNative FileSystem (WNFS) specification. WNFS is a versioned content-addressable distributed filesystem with private and public sub systems. The private filesystem is encrypted so that only users with the right keys can access its contents. It is designed to prevent inferring metadata like the structure of the file tree. The other part of the WNFS filesystem is a simpler public filesystem that is not encrypted and can be accessed by anyone with the right address.

WNFS also features collaborative editing of file trees, where multiple users can edit the same tree at the same time.

WNFS file trees can serialize and be deserialized from IPLD graphs with an extensible metadata section. This allows WNFS to be understood by other IPLD-based tools and systems.

This library is designed with WebAssembly in mind. You can follow instructions on how to use it in your browser applications here.

Outline

Crates

This is the dependency graph between these crates:

flowchart TD
    wnfs-wasm --> wnfs
    wnfs-wasm --> wnfs-nameaccumulator
    %% wnfs-bench --> wnfs
    %% wnfs-bench --> wnfs-hamt
    %% wnfs-bench --> wnfs-nameaccumulator
    wnfs --> wnfs-hamt
    wnfs --> wnfs-common
    wnfs --> wnfs-unixfs-file
    wnfs --> wnfs-nameaccumulator
    wnfs-unixfs-file --> wnfs-common
    wnfs-hamt --> wnfs-common
    wnfs-nameaccumulator --> wnfs-common

Building the Project

REQUIREMENTS

  • The Rust Toolchain

Follow the instructions here to install the official Rust toolchain.

  • The WebAssembly Toolchain

If you are interested in compiling the project for WebAssembly, you can follow the instructions below.

<summary>Read more</summary>
  • Install wasm32-unknown-unknown target

    rustup target add wasm32-unknown-unknown
    
  • rust-analyzer is the go-to IDE tool for Rust and if you have it set up, you may want to set the rust-analyzer.cargo.target setting to wasm32-unknown-unknown

  • Install wasm-bindgen

    cargo install wasm-bindgen-cli
    
  • Install playwright binaries

    npx playwright install
    

Architecture-specifics

  • On ARM-based (M1 family) macOS, you might need to explicitly install the following:

    • Install wasm-bindgen

      cargo install -f wasm-bindgen-cli
      
    • Install wasm-opt

      brew install binaryen
      
  • On Arch Linux based distributions, you might need to explicitly install the following:

    • Install wasm-opt

      sudo pacman -S binaryen
      

  • The rs-wnfs Command

You can optionally set up the rs-wnfs script.

<summary>Read more</summary>
  • Install it using the following command:

    sh ./scripts/rs-wnfs.sh setup
    
  • This lets you run the rs-wnfs.sh script as a command.

    rs-wnfs help
    

STEPS

  • Clone the repository.

    git clone https://github.com/wnfs-wg/rs-wnfs.git
    
  • Change directory

    cd rs-wnfs
    
  • Build the project

Check REQUIREMENTS on how to set up the rs-wnfs command.

  scripts/rs-wnfs build
  • You can also build for specific crates

    scripts/rs-wnfs build --wasm
    

Usage

WNFS does not have an opinion on where you want to persist your content or the file tree. Instead, the API takes any object that implements the asynchronous BlockStore trait. The library also avoids including system function calls that could possibly tie it to a set of platforms. Operations like time and random number generation have to be passed in via the API. This allows the library to be used in a wide variety of environments. It particularly makes virtualisation easier.

Let's see an example of working with a public filesystem. We will use the in-memory block store provided by the library.

use anyhow::Result;
use chrono::Utc;
use wnfs::{
    common::MemoryBlockStore,
    public::PublicDirectory
};

#[async_std::main]
async fn main() -> Result<()> {
    // Create a new public directory.
    let dir = &mut PublicDirectory::new_rc(Utc::now());

    // Create an in-memory block store.
    let store = &MemoryBlockStore::default();

    // Add a /pictures/cats subdirectory.
    dir.mkdir(&["pictures".into(), "cats".into()], Utc::now(), store)
        .await?;

    // Store the the file tree in the in-memory block store.
    dir.store(store).await?;

    // List all files in /pictures directory.
    let result = dir.ls(&["pictures".into()], store).await?;

    println!("Files in /pictures: {:#?}", result);

    Ok(())
}

Here we create a root directory dir and subsequently add a /pictures/cats subdirectory to it. As mentioned earlier, system-level operations like time are passed in from the API. In this case, we use the Utc::now() function from the [chrono][chrono-crate] crate to get the current time.

PublicDirectory gets wrapped in Rc here because it lets us pass it around without worrying about ownership and lifetimes. Making the Rc &mut futher allows us to relinquish ownership to the interior PublicDirectory and point to a new one when needed (essentially for every write). This immutable way of handling changes has cool benefits like tracking and rolling back changes. It also makes collaborative editing easier to implement and reason about. You can find more examples in the wnfs/examples/ folder.

That's the public filesystem, the private filesystem, on the other hand, is a bit more involved. The Hash Array Mapped Trie (HAMT) is where we store the private filesystem tree and some other information related to it. HAMT allows for effective storage and retrieval of encrypted and obfuscated filesystem trees and PrivateForest is basically a HAMT that can contain multiple file trees with hash for keys and CIDs for values.

use anyhow::Result;
use chrono::Utc;
use rand_chacha::ChaCha12Rng;
use rand_core::SeedableRng;
use wnfs::{
    common::MemoryBlockStore,
    private::{
        PrivateDirectory,
        forest::{hamt::HamtForest, traits::PrivateForest},
    }
};

#[async_std::main]
async fn main() -> Result<()> {
    // Create an in-memory block store.
    let store = &MemoryBlockStore::default();

    // A random number generator.
    let rng = &mut ChaCha12Rng::from_entropy();

    // Create a private forest.
    let forest = &mut HamtForest::new_trusted_rc(rng);

    // Create a new private directory.
    let dir = &mut PrivateDirectory::new_rc(&forest.empty_name(), Utc::now(), rng);

    // Add a file to /pictures/cats directory.
    dir.mkdir(
        &["pictures".into(), "cats".into()],
        true,
        Utc::now(),
        forest,
        store,
        rng,
    )
    .await?;

    // Add a file to /pictures/dogs/billie.jpg file.
    dir.write(
        &["pictures".into(), "dogs".into(), "billie.jpg".into()],
        true,
        Utc::now(),
        b"Hello! This is billie".to_vec(),
        forest,
        store,
        rng,
    )
    .await?;

    // List all files in /pictures directory.
    let result = dir.ls(&["pictures".into()], true, forest, store).await?;

    println!("Files in /pictures: {:#?}", result);

    Ok(())
}

This example introduces a few new concepts. The first is the HamtForest which is a HAMT that can contain multiple file trees and implements the PrivateForest interface needed for persisting private file systems.

The second is the Name (returned from forest.empty_name()) and NameAccumulator that lets us identify nodes in the filesystem, and are suitable for offspring proving.

Finally, we have the random number generator, rng, that the library uses for generating new keys and other random values needed for the protocol.

Check the wnfs/examples/ folder for more examples.

Testing the Project

  • Run all tests

    scripts/rs-wnfs test
    
  • Run benchmarks

    scripts/rs-wnfs bench
    

You can also find a nice graph of the CI benchmarks here.

Contributing

Pre-commit Hook

This library recommends using pre-commit for running pre-commit hooks. Please run this before every commit and/or push.

  • Once installed, Run pre-commit install to setup the pre-commit hooks locally. This will reduce failed CI builds.
  • If you are doing interim commits locally, and for some reason if you don't want pre-commit hooks to fire, you can run git commit -a -m "Your message here" --no-verify.

Conventional Commits

This project lightly follows the Conventional Commits convention to help explain commit history and tie in with our release process. The full specification can be found here. We recommend prefixing your commits with a type of fix, feat, docs, ci, refactor, etc..., structured like so:

<type>[optional scope]: <description>

[optional body]

[optional footer(s)]

Getting Help

For usage questions, usecases, or issues reach out to us in our Discord webnative-fs channel. We would be happy to try to answer your question or try opening a new issue on Github.

License

This project is licensed under the Apache License 2.0.