attachments small fix + added some documentation

README.md

@@ -1,75 +1,57 @@
 🦀Crabmail🦀
 ===========
 
-[self-hosted](https://git.alexwennerberg.com/crabmail/) / [github mirror](https://github.com/alexwennerberg/crabmail)
-
-A static mail HTML and archive for
-the 21st century, written in Rust. Includes helpful "modern" features that
-existing solutions lack, like:
-
-* Responsive UI
-* Single-page threads
-* Working mailto: links
-* Thread-based Atom feeds
-
-Not implemented yet / designed:
-* Attachment handling?
-
-EMAIL FOREVER!
-
 Installation and usage
 ----------------------
 
-To install:
+Clone the project:
+```bash
+git clone https://git.42d.io/reda/crabmail
 ```
-git clone git://git.alexwennerberg.com/crabmail/
+Install dependencies:
+```bash
 cd crabmail && cargo install --path .
 ```
+Add WASM target for cargo toolchain:
+```bash
+rustup target add wasm32-wasi
+```
+For Linux environment build and run:
+```bash
+cargo run --release
+```
+For WASI environment build:
+```bash
+cargo build --target wasm32-wasi --release
+```
+For WASI environment run:
+Note: To give WASI access to a directory, you need to use the --dir flag and specify the path in the {actual_path}:{virtual_path} format, where {actual_path} should be replaced with the actual path in the local file system, and {virtual_path} should represent the virtual path that WASI will use in the code.
+```bash
+wasmedge --dir {path with config} --dir {path to store results} --dir {path with maildir} {path to .wasm file} -h -c {path to the config file} -d {path to the result dir} -m {path to maildir}
+```
+For WASI environment run example:
+```
+wasmedge --dir /home/yura/crabmailConfig/:/home/yura/crabmailConfig/ --dir /home/yura/crabmailResult/:/home/yura/crabmailResult/ --dir /home/yura/lists/:/home/yura/lists/ ./target/wasm32-wasi/release/crabmail.wasm -h -c /home/yura/crabmailConfig/crabmail.conf -d /home/yura/crabmailResult -m /home/yura/lists/
+```
 
 Copy `crabmail.conf` and set the variables as needed.
 
-Get a maildir folder, for example, via `mbsync`. Crabmail will create sub-lists
-for each folder inside this maildir.
+Sonic search
+-----
 
-Run crabmail [maildir root] -c [config-file.conf].
+For Sonic Search to work, a Sonic [server](https://github.com/valeriansaliou/sonic) must be running in the background. Additionally, ensure that IPv6 is not used with WASI, as it is not yet supported.
 
-For more thorough documentation, run `man doc/crabmail.1`. You can also move
-these wherever your docs manpages may live
-
-If you want to use an mbox file (for example, to mirror another archive), use
-[mblaze](https://github.com/leahneukirchen/mblaze) to import it into a maildir.
-Mblaze also has some tools that you may find supplementary to crabmail.
-
-For example:
-```
-mkdir -p lists/mylist/cur lists/mylist/tmp lists/mylist/new
-mdeliver -M lists/mylist < mylist.mbox
-crabmail lists
-```
-
-Open `site/index.html` in a web browser 
+Iroh sync
+-----
+This is a separate [module](https://git.42d.io/yurii/Iroh_sync)
 
 NOTES
 -----
 
-This is only tested on Linux. Notably: Any character other than "/" is allowed
-in filenames used in message-id. Make sure this doesn't break anything or cause
-a security vuln on your filesystem.
-
-Contributing
-------------
-
-For patches, use `git-send-email` or `git-format-patch`
-to send a patch to my [mailing list](https://lists.sr.ht/~aw/patches)
-
-`git-format-patch` is preferred for non-trivial or multi-commit changes
+The program works on Linux and WASI. You need to use Wasmedge to run WASM code because the project uses libraries from Wasmedge for networking and asynchronous tasks 
 
 Etc
 ---
 
 Crabmail is AGPLv3 licenses, but some files are licensed under 0BSD or other
 more permissive licenses. I call this out when I can.
-
-For a similar project, check out [bubger](https://git.causal.agency/bubger/about/)
-
-Consider supporting me and my projects on [patreon](https://www.patreon.com/alexwennerberg)

src/imap.rs

@@ -30,6 +30,7 @@ use async_imap_wasi::{Client, Session};
 #[cfg(target_os = "wasi")]
 use async_imap_wasi::extensions::idle::IdleResponse::NewData;
 
+/// create TLS connect with the IMAP server
 pub async fn connect_to_imap() -> anyhow::Result<Client<TlsStream<TcpStream>>>{
     let mut root_cert_store = RootCertStore::empty();
     root_cert_store.extend(webpki_roots::TLS_SERVER_ROOTS.iter().cloned());
@@ -50,6 +51,7 @@ pub async fn connect_to_imap() -> anyhow::Result<Client<TlsStream<TcpStream>>>{
     Ok(client)
 }
 
+/// download all emails from the server and persist them
 pub async fn download_email_from_imap() -> anyhow::Result<Vec<(u32, PathBuf)>>{
     let mut client = connect_to_imap().await?;
     let mut session_result = client.login(Config::global().username.clone(), Config::global().password.clone()).await;
@@ -85,6 +87,7 @@ pub async fn download_email_from_imap() -> anyhow::Result<Vec<(u32, PathBuf)>>{
     Ok(stored_paths)
 }
 
+/// return Vec with all mailboxes
 pub async fn list_imap_folders(session: &mut Session<TlsStream<TcpStream>>) -> anyhow::Result<Vec<String>> {
     let mut folders: Vec<String> = vec![];
     let mut folders_stream = session.list(None, Some("*")).await?;
@@ -99,6 +102,7 @@ pub async fn list_imap_folders(session: &mut Session<TlsStream<TcpStream>>) -> a
     Ok(folders)
 }
 
+/// download all emails from one mailbox 
 pub async fn fetch_and_store_emails(session: &mut Session<TlsStream<TcpStream>>, list: String) -> anyhow::Result<Vec<(u32, PathBuf)>> {
     session.select(list.clone()).await?;
 
@@ -161,6 +165,7 @@ pub async fn fetch_and_store_emails(session: &mut Session<TlsStream<TcpStream>>,
     Ok(stored_paths)
 }
 
+/// read uids that have been already downloaded
 fn read_local_uids(uids_path: PathBuf) -> anyhow::Result<HashMap<String, HashSet<String>>> {
     let file = match File::open(&uids_path) {
         Ok(file) => file,
@@ -191,6 +196,7 @@ fn read_local_uids(uids_path: PathBuf) -> anyhow::Result<HashMap<String, HashSet
     Ok(result)
 }
 
+/// add uid to the local list of downloaded uids
 fn update_local_uids(uid: String, list: String, uids_path: PathBuf) -> anyhow::Result<()> {
     let file = std::fs::OpenOptions::new()
         .write(true)
@@ -217,7 +223,9 @@ fn update_local_uids(uid: String, list: String, uids_path: PathBuf) -> anyhow::R
     Ok(())
 }
 
+/// delete single message remotely by uid (marks deleted and deletes from the server at the same time)
 pub async fn delete_email_by_uid(list: String, uid: u32) -> anyhow::Result<()>{
+    // TODO split to mark_deleted() and expunge()
     let mut client = connect_to_imap().await?;
     let mut session_result = client.login(Config::global().username.clone(), Config::global().password.clone()).await;
     let mut session = match session_result {
@@ -226,7 +234,6 @@ pub async fn delete_email_by_uid(list: String, uid: u32) -> anyhow::Result<()>{
     };
     
     session.select(list.clone()).await?;
-    // session.mark_deleted(uid).await?;
     let updates_stream = session.store(format!("{}", uid), "+FLAGS (\\Deleted)").await?;
     let _updates: Vec<_> = updates_stream.try_collect().await?;
     let _stream = session.expunge().await?;
@@ -235,6 +242,7 @@ pub async fn delete_email_by_uid(list: String, uid: u32) -> anyhow::Result<()>{
     Ok(())
 }
 
+/// create a folder remotely
 pub async fn create_folder(name: String) -> anyhow::Result<()>{
     let mut client = connect_to_imap().await?;
     let mut session_result = client.login(Config::global().username.clone(), Config::global().password.clone()).await;
@@ -247,6 +255,7 @@ pub async fn create_folder(name: String) -> anyhow::Result<()>{
     Ok(())
 }
 
+/// rename a folder remotely
 pub async fn rename_folder(name: String, new_name: String) -> anyhow::Result<()>{
     let mut client = connect_to_imap().await?;
     let mut session_result = client.login(Config::global().username.clone(), Config::global().password.clone()).await;
@@ -259,6 +268,7 @@ pub async fn rename_folder(name: String, new_name: String) -> anyhow::Result<()>
     Ok(())
 }
 
+/// delete a folder remotely
 pub async fn delete_folder(name: String) -> anyhow::Result<()>{
     let mut client = connect_to_imap().await?;
     let mut session_result = client.login(Config::global().username.clone(), Config::global().password.clone()).await;
@@ -271,6 +281,7 @@ pub async fn delete_folder(name: String) -> anyhow::Result<()>{
     Ok(())
 }
 
+/// run a async task to monitor any changes in the mailbox 
 pub async fn check_for_updates(mailbox: String) -> anyhow::Result<()> {
     task::spawn(async move {
         let mut client = match connect_to_imap().await {
@@ -296,13 +307,14 @@ pub async fn check_for_updates(mailbox: String) -> anyhow::Result<()> {
             idle.init().await.unwrap();
             let (idle_wait, interrupt) = idle.wait();
             let idle_handle = task::spawn(async move {
-                println!("IDLE: waiting for 30s");
+                println!("IDLE: waiting for 30s"); // TODO remove debug prints
                 sleep(Duration::from_secs(30)).await;
                 println!("IDLE: waited 30 secs, now interrupting idle");
                 drop(interrupt);
             });
 
             match idle_wait.await.unwrap() {
+                // TODO add more cases like delete, move...
                 NewData(data) => {
                     // TODO do not update all emails (IMAP returns * {number} RECENT) and do it only for one mailbox
                     let new_paths = download_email_from_imap().await.expect("Cannot download new emails");
@@ -329,6 +341,7 @@ pub async fn check_for_updates(mailbox: String) -> anyhow::Result<()> {
     Ok(())
 }
 
+/// saves a raw email properly
 fn store(path: PathBuf, uid: String, subfolder: String, data: &[u8], info: &str) -> io::Result<PathBuf> {
     // loop when conflicting filenames occur, as described at
     // http://www.courier-mta.org/maildir.html

src/indexes.rs

@@ -39,6 +39,7 @@ impl PartialEq for SerializableMessage {
             self.date == other.date &&
             self.uid == other.uid &&
             self.list == other.list
+        // TODO add more properties
     }
 }
 

src/main.rs

@@ -13,17 +13,16 @@ use serde::{Deserialize, Serialize};
 use crate::indexes::{Indexes};
 use config::{Config, INSTANCE};
 use crate::server::{run_api};
-use mailparse::parse_mail;
+use mailparse::{DispositionType, parse_mail, ParsedMail};
 use regex::Regex;
 use crate::templates::util::parse_email;
 use crate::util::{compress_and_save_file};
 use crate::imap::{delete_folder, rename_folder, create_folder, check_for_updates};
 use crate::js::email_scripts;
 use warp::Filter;
-use crate::sonic::{IngestSonic, SearchSonic};
+use crate::sonic::{IngestSonic};
 
 mod smtp_client;
-
 mod arg;
 mod config;
 mod models;
@@ -35,6 +34,7 @@ mod server;
 mod js;
 mod sonic;
 
+/// appends an extension to a path
 pub fn append_ext(ext: impl AsRef<OsStr>, path: &PathBuf) -> PathBuf {
     let mut os_string: OsString = path.into();
     os_string.push(".");
@@ -42,17 +42,7 @@ pub fn append_ext(ext: impl AsRef<OsStr>, path: &PathBuf) -> PathBuf {
     os_string.into()
 }
 
-fn write_if_changed<T: AsRef<[u8]>>(path: &PathBuf, data: T) -> bool {
-    if let Ok(d) = std::fs::read(path) {
-        if d == data.as_ref() {
-            return false;
-        }
-    }
-
-    std::fs::write(path, data).unwrap();
-    true
-}
-
+/// converts message to HTML string
 pub fn message_to_html(msg: StrMessage) -> String{
     fn remove_style_tags(html: &str) -> String {
         Regex::new(r"(?s)<style.*?>.*?</style>").unwrap()
@@ -101,6 +91,10 @@ pub fn message_to_html(msg: StrMessage) -> String{
     let mut body_content = remove_style_tags(&extract_body(&html));
     // body_content = purple_numbers(&*body_content, "#");
 
+    // retrieving attachments
+    retrieve_and_save_attachments(parsed_mail, msg.list.clone(), msg.clone().hash)
+        .expect("Unable to retrieve attachment");
+    
     format!(
         r#"
                 <!DOCTYPE html>
@@ -129,9 +123,37 @@ pub fn message_to_html(msg: StrMessage) -> String{
     )
 }
 
+fn retrieve_and_save_attachments(parsed_mail: ParsedMail, list: String, file_name: String) -> anyhow::Result<()>{
+    // iterating through each part of email
+    for part in parsed_mail.subparts {
+        let disposition = part.get_content_disposition();
+        // if this is an attachment
+        if disposition.disposition == DispositionType::Attachment {
+            // creating directory for it
+            let full_dir = Config::global().out_dir.clone()
+                .join(list.clone())
+                .join(".attachments")
+                .join(file_name.clone());
+            create_dir_all(full_dir.clone()).expect("Unable to create directory for attachments");
+
+            let mut filename = disposition.params.get("filename")
+                .map(|s| s.to_string())
+                .unwrap_or_else(|| "unnamed_file".to_string());
+
+            // creating full absolute path to the file
+            let full_path = full_dir.join(filename.clone());
+            let mut output = File::create(full_path.clone())?;
+
+            // writing the data to the file
+            output.write_all(&part.get_body_raw().unwrap())?;
+        }
+    }
+
+    Ok(())
+}
+
+/// saves email to local file system
 fn persist_email(msg: &Message, uid: u32, list: String, original_path: PathBuf) -> anyhow::Result<()>{
-    let mut indexes = Indexes::new();
-    
     // creating all folders
     if Config::global().out_dir.exists() {
         std::fs::create_dir_all(&Config::global().out_dir).ok();
@@ -162,18 +184,11 @@ fn persist_email(msg: &Message, uid: u32, list: String, original_path: PathBuf)
             Err(_) => {println!("Error while compressing or saving {}", message.hash.clone())}
         };
     }
-
-    // // xml
-    // let xml = append_ext("xml", &message_xml_dir.join(message.hash.clone()));
-    // let mut file = File::create(xml)?;
-    // let data = message.to_xml();
-    // file.write_all(data.as_bytes())?;
-    // file.flush()?;
     
-    // sonic
+    // ingest text from the message to the sonic server // TODO create as queue if server is not accessible
     let mut ingest_channel = IngestSonic::new()?;
     let data = message.to_ingest();
-    match ingest_channel.ingest_document("emails", &*list, &*message.hash.clone(), &*data) { // message.hash to id
+    match ingest_channel.ingest_document("emails", &*list, &*message.hash.clone(), &*data) { // TODO change message.hash to id
         Ok(_) => {}
         Err(_) => {println!("Unable to ingest {} email to sonic search", message.hash.clone())}
     };
@@ -186,6 +201,7 @@ fn persist_email(msg: &Message, uid: u32, list: String, original_path: PathBuf)
     Ok(())
 }
 
+/// read raw email and saves HTML version
 pub fn add_email(path: PathBuf, uid: u32) -> anyhow::Result<()>{
     let maildir = match path.parent() {
         None => {return Err(anyhow!("Cannot retrieve parent folder from the path"))}
@@ -210,6 +226,7 @@ pub fn add_email(path: PathBuf, uid: u32) -> anyhow::Result<()>{
     Ok(())
 }
 
+/// delete a single message both locally and remotely
 async fn delete_email(message_id: String) -> anyhow::Result<()>{
     let message = Indexes::delete_from_messages(message_id);
     match message {
@@ -241,7 +258,7 @@ async fn delete_email(message_id: String) -> anyhow::Result<()>{
     Ok(())
 }
 
-// create folder both locally and remotely
+/// create folder both locally and remotely
 pub async fn create_folder_lar(name: String) -> anyhow::Result<()>{
     match create_folder(name.clone()).await {
         Ok(_) => {
@@ -258,7 +275,7 @@ pub async fn create_folder_lar(name: String) -> anyhow::Result<()>{
     }
 }
 
-// delete folder both locally and remotely
+/// delete folder both locally and remotely
 pub async fn delete_folder_lar(name: String) -> anyhow::Result<()>{
     match delete_folder(name.clone()).await {
         Ok(_) => {
@@ -275,7 +292,7 @@ pub async fn delete_folder_lar(name: String) -> anyhow::Result<()>{
     }
 }
 
-// rename folder both locally and remotely
+/// rename folder both locally and remotely
 pub async fn rename_folder_lar(old_name: String, new_name: String) -> anyhow::Result<()>{
     match rename_folder(old_name.clone(), new_name.clone()).await {
         Ok(_) => {
@@ -293,6 +310,7 @@ pub async fn rename_folder_lar(old_name: String, new_name: String) -> anyhow::Re
     }
 }
 
+/// Parse args from CLI
 fn parse_args(){
     let args = arg::Args::from_env();
     
@@ -306,11 +324,11 @@ fn parse_args(){
 }
 
 async fn run(){
+    // parse args from CLI
     parse_args();
 
     // downloading new emails
     let new_paths = imap::download_email_from_imap().await.expect("Cannot download new emails");
-
     
     let mut lists: Vec<String> = Vec::new();
 
@@ -347,8 +365,17 @@ async fn run(){
         Indexes::persist_threads().expect("Unable to persist threads");
         Indexes::persist_indexes(lists).expect("Unable to persist indexes");    
     }
+
+    // monitor updates in INBOX
+    if let Err(e) = check_for_updates("INBOX".to_string()).await {
+        eprintln!("Failed to monitor mailbox: {:?}", e);
+    }
+
+    // API
+    run_api().await;
 }
 
+/// Entry point for a wasi env
 #[cfg(target_os = "wasi")]
 #[tokio::main(flavor = "current_thread")]
 async fn main() -> anyhow::Result<()> {
@@ -377,31 +404,13 @@ async fn main() -> anyhow::Result<()> {
     
     // delete_email("Sent".to_string(), 4).await;
     
-    // imap::create_folder("testFolder".to_string()).await.unwrap();
-    // imap::rename_folder("testFolder".to_string(), "newTestFolder".to_string()).await.unwrap();
-    // imap::delete_folder("newTestFolder".to_string()).await.unwrap();
-    
-    
-    // looking for updates
-    if let Err(e) = check_for_updates("INBOX".to_string()).await {
-        eprintln!("Failed to monitor mailbox: {:?}", e);
-    }    
-    
-    let duration = start.elapsed();
-    println!("Duration {:?}", duration);
-    
-    
-    // API
-    run_api().await;
-    
     Ok(())
 }
 
+/// Entry point for a not wasi env
 #[cfg(not(target_os = "wasi"))]
 #[tokio::main]
 async fn main() {
-    let start = Instant::now();
-    
     run().await;
     
     // let email = LettreMessage::builder()
@@ -424,15 +433,4 @@ async fn main() {
 
     
     // delete_email("lqo7m8r2.1js7w080jvr2o@express.medallia.com".to_string()).await.expect("Unable to delete an email");
-
-    
-    if let Err(e) = check_for_updates("INBOX".to_string()).await {
-        eprintln!("Failed to monitor mailbox: {:?}", e);
-    }
-
-    let duration = start.elapsed();
-    println!("Duration {:?}", duration);
-    
-    // API
-    run_api().await;
 }